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ABSTRACT 


An  introductory,  single-degree-of- freedom  experimental 
study  of  aircraft  roll  control  was  performed  to  demonstrate  the 
feasibility  of  doing  flying  qualities  research  via  direct 
manipulation  of  an  aircraft's  step  response  on  a  flight  simulator. 
This  is  a  very  attractive  approach  for  studying  the  flying  qualities 
requirements  of  modern  aircraft  that  will  incorporate  new  and 
sophisticated  digital  fly-by-wire  control  schemes  for  coordinating 
all  control  forces  to  produce  nonclassical  responses  to  pilot 
commands.  Central  to  the  work  was  the  development  of  a  digital 
simulation  technique,  based  on  the  mathematical  concept  of 
convolution,  that  provides  the  means  for  easy  manipulation  of  the 
simulated  vehicle's  step-response  characteristics.  Two  experiments 
were  performed  that  examined  several  basic  characteristics  of 
an  aircraft' s  roll-rate  step-response  in  a  simulated  tracking 
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1.  INTRODUCTION 


Flight  control  systems  of  modern,  high-performance  military 
aircraft  are  on  the  brink  of  marked  change  brought  about  by  the 
development  of  the  onboard  computer  into  a  powerful,  highly 
reliable,  lightweight,  and  relatively  inexpensive  tool.  With 
it  as  a  primary  system,  dramatic  new  control  concepts  are 
realizable.  Control  configured  vehicle  designs  can  reduce 
aircraft  weight  and  drag  by  "trading"  fixed  aerodynamic  surfaces 
for  artificial  stabilization.  Aircraft  mechanical  complexity 
can  be  greatly  reduced  by  the  use  of  wholly  electronic  fly-by¬ 
wire  control  systems.  And,  of  particular  interest  here,  it 
becomes  possible  to  use  model  reference  algorithms  or  other 
advanced,  real  time  computational  schemes  to  produce  fly-by-wire 
commands.  In  this  way  all  controls  can  be  manipulated  in  what¬ 
ever  ways  —  no  matter  how  complicated  —  that  might  be  necessary 
to  produce  "ideal"  aircraft  responses  at  every  flight  condition. 

To  exploit  the  new  capabilities  fully,  the  control  system 
designer  needs  better  knowledge  of  what  the  pilot's  preferences 
really  are.  This  is  because  current  flying  qualities  specifications 
and  most  flying  qualities  research  start  with  the  premise  that 
aircraft  behave  in  stereotyped  ways.  Implementation  of  the  new 
control  concepts  can,  however,  change  this  markedly  and  make  it 
possible  for  aircraft  to  respond  to  pilot  commands  in  almost  any 
way  desired.  Thus  the  old  rules  no  longer  apply,  and  the  designer 
needs  new  guidelines  for  using  the  unprecedented  freedom  he  will 
have  in  choosing  how  a  vehicle  responds  to  control  inputs. 

Current  flying  qualities  research  at  Grumman  is  using  some 
novel  experimental  simulation  techniques  in  an  attempt  to  uncover 
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basic  pilot  preferences  and  to  provide  the  flying  qualities 
specifications  for  the  next  generation  of  aircraft  control 
systems.  The  work  starts  with  the  belief  that  more  meaningful 
flying  qualities  experimentation  is  possible  when  the  experimenter 
has  direct  control  over  those  elements  of  vehicle  motion  that  the 
pilot  senses.  Conventional  system  describers  like  pole  and  zero 
locations  (though  convenient  to  the  dynamicist)  are  not  what 
a  pilot  feels,  and  therefore  probably  are  not  the  elements  of 
motion  that  shape  his  basic  preferences.  Thus,  in  the  present 
work,  conventional  analytic  describers  are  abandoned,  and 
vehicle  dynamics  are  described  by  the  shape  of  the  time  response 
to  a  unit  step  input  of  pilot  control.  For  linear  systems,  the 
step  response  is  a  unique  description  of  vehicle  dynamics,  and 
its  shape  graphically  portrays  the  elements  of  motion  that  a 
pilot  feels  in  direct  response  to  his  control  actions.  Thus, 
by  manipulating  the  shape  of  the  step  response  the  experimenter 
can  alter  vehicle  dynamics  by  changing  those  elements  of  vehicle 
motion  that  the  pilot  senses  directly.  In  addition,  the  step 
response  is  a  complete  description  of  vehicle  dynamics.  It 
could  represent  the  only  item  that  would  have  to  be  checked  for 
compliance  to  a  specification,  as  it  includes  the  effects  of 
linkages,  actuator  performance,  computational  algorithms,  and 
all  other  aspects  of  the  control  system  between  the  pilot' s 
command  and  the  vehicle's  response.  Finally,  as  a  time  domain 
quantity,  it  is  a  better  descriptor  for  the  modern  computational 
schemes  used  to  produce  "model  reference"  or  "ideal"  control 
responses . 

The  work  described  here  respresents  a  first  tentative  step 
in  a  long-range  program  of  experimental  research  designed  to 
determine,  from  the  pilot's  perceptual  viewpoint,  what  the  basic 


elements  of  a  vehicle' s  step  response  are  and  how  various 
combinations  of  them  affect  flyability.  It  is  concerned  primarily 
with  the  development  of  a  technique  for  the  ready  manipulation  of 
the  step  response  of  simulated  vehicles  and  the  demonstration  of 
this  technique  as  a  tool  for  studying  aircraft  flying  qualities. 
Toward  this  end  a  series  of  single-degree-of-freedom  roll-tracking 
experiments  were  performed  in  which  slopes,  radii,  and  other 
graphical  characteristics  of  the  roll-rate  step-response  were 
the  independent  variables.  The  final  goal  of  the  program  is  to 
produce  an  understanding  of  aircraft  control  responses  —  well 
beyond  the  restrictions  of  old  conventional  aircraft  stereotypes  — 
that  could  be  used  to  develop  design  guides  for  the  next  generation 
of  highly  augmented  craft. 

In  this  report,  the  Simulation  Technique  section  covers  the 
development  of  the  computational  algorithm  for  driving  a 
simulator  in  a  way  that  allows  facile  manipulation  of  the  step 
response  of  the  simulated  vehicle.  The  Experimentation  section 
includes  development  of  the  "pair  comparisons"  experimental 
technique.  This  procedure  was  instituted  to  accommodate  the 
non-pilots  that  were  used  as  subjects.  And,  lastly,  the  success 
of  the  methods  used  are  demonstrated  in  the  Discussion  of  Results 
section  where  the  outcome  of  the  roll-control  experiments  are 
presented. 
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2.  SIMULATION  TECHNIQUE 


The  central  feature  of  the  simulator  studies  discussed  here 
is  the  computational  algorithm  employed  to  generate  the  simulator 
drive  signals  on  a  digital  computer  and  the  reasons  why  it  was 
used.  Classically,  the  dynamic  behavior  of  a  system  is  character¬ 
ized  by  a  differential  equation  and  analyzed  in  the  frequency 
domain  via  the  Laplace  transform.  These  concepts  have  been 
molded  by  generations  of  mathematicians,  physicists,  and  engineers 
into  extraordinarily  powerful  tools  for  insuring  the  successful 
operation  of  complex  mechanisms.  However,  they  have  two  major 
deficiencies  when  employed  as  system  describers  for  flying-qualities 
research.  First,  when  a  pilot  assumes  control  of  an  aircraft 
system,  he  becomes  the  final  arbiter  of  its  worth,  and  the 
researcher  should  therefore  define  system  behavior  in  terms  of 
elements  that  the  pilot  can  relate  to  directly.  The  parameters 
of  a  differential  equation  or  transfer  function,  for  instance, 
bear  no  direct,  simple  relationship  to  the  sensations  of  the 
operator,  and  it  becomes  difficult  to  establish  correlation  between 
pilot  opinion  or  performance  and  the  value  of  a  parameter.  A 
vehicle  pilot  does  not  directly  sense  and  appreciate  things  like 
frequency,  damping  ratio,  pole/zero  location,  or  a  stability 
derivative,  but  rather  things  like  force  on  his  body  and  cockpit 
motion,  varying  in  response  to  his  own  commands.  Thus,  an 
experiment  to  define  and  quantify  the  dynamic  elements  recognized 
by  the  pilot  ought  to  be  done  in  the  pilot's  sensory  domain: 
cockpit  kinematics. 

The  other  major  deficiency  of  conventional  analytic  techniques, 
and  the  one  of  most  concern  here,  is  that  they  can  become 
extremely  complex  when  used  to  describe  modern,  highly  augmented 
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aircraft  systems.  To  make  a  general  study  of  the  flying  qualities 
of  such  systems  requires  the  consideration  of  a  very  large  number 
of  parameters  which,  as  cited  above,  bear  no  simple  relationship 
to  the  sensations  of  the  pilot.  Even  more  discouraging  is  the 
fact  that  conventional  analytic  descriptors  are  very  cumbersome 
when  it  comes  to  varying  some  of  the  simpler,  more  basic  elements 
of  cockpit  kinematics.  To  make  an  independent  variation  of  one 
characteristic  of  cockpit  motion  frequently  requires  the 
complicated  adjustment  of  all  the  parameters  used  to  describe  the 
high-order  augmented  system.  What  is  needed  is  a  way  to  characterize 
a  system  by  its  kinematics  alone  by  a  "motion  signature"  that 
defines  it  exclusively  and  directly  according  to  the  way  it 
moves  in  response  to  a  command. 

Convolution  is  an  old  mathematical  process  adaptable  to 
this  problem.  If  a  system-variable’ s  response  to  a  particular 
kind  of  input  —  a  step  —  is  known,  the  convolution  algorithm 
describes  how  that  variable  responds  to  any  input.  In  other  words, 
the  behavior  of  the  variable  is  completely  defined  by  its  step- 
response.  The  step-response  becomes  the  desired  "mot ion- signature" 
of  the  system,  and  no  other  descriptors  are  needed.  Thus  for 
flying  qualities  research  the  shape  of  the  step-response  becomes 
the  independent  variable.  What  we  must  do  is  mechanize  the 
convolution  integral  for  "real-time"  solution  on  a  digital  computer 
in  a  way  that  allows  easy,  direct  manipulation  of  the  step  response. 
Convolution  is  of  course  a  linear,  time-invariant  process  that 
must  start  with  no  initial  conditions  on  the  output,  but  this 
is  not  a  limitation  of  any  consequence  in  the  present  application. 

We  want  to  study  the  step-response  in  many  separate  and  fixed 
vehicle/task  situations  to  learn  what  is  the  best  for  each 
one,  and  the  simulation  can  always  start  from  zero  initial 
conditions.  Some  nonlinearities  that  are  of  interest 
are  things  like  deadzone ,  which  can  be  simulated  by 
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preconditioning  the  command  signal  (stick  position)  and  stiction 
or  friction,  which  would  have  to  be  applied  mechanically  to  the 
simulator' s  control  lever. 


The  convolution  integral  can  be  written  as 

G (t)  =  F(t)  H(o)  +  J  F(t)  H(t-T)dx _ 

o 


where 


H  is  the  step  response  *-  the  "motion  signature" 
• 

H  is  the  impulse  response 

F  is  the  input  or  "command" 

G  is  the  systems  response  to  the  command 


(1) 


The  sample-data  form  used  with  uniform  sampling  for  solution  on  a 
digital  computer  becomes 

K=N 

GH  =  FNHo  +  Z  FK  F‘N-K  At 
K=0 


where  F^  =  F(K  At) 

Hr  =  H(K  At) 

At  =  the  sample  interval 
N  =  the  greatest  integer  <  t/AT 


For  the  applications  of  interest  here,  the  step  response  always 
starts  at  zero;  that  is  Hq  =  0.  Thus  Eq.  (2)  can  be  simplified  to 

GN  =  V  FKHN_K  At  (3) 

tfco 


This  is  the  form  used  to  compute  system  output  in  response  to 
a  pilot  input  forcing  function  in  single-degree-of-freedom  simulator 
tracking  studies.  Note  that  H  must  go  to  zero  in  a  finite  number 
of  samples  so  that  it  can  be  stored  in  a  finite  length  buffer. 
However,  included  in  this  class  of  response  are  all  the  responses  of 
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interest  here  .  Although  the  impulse  response,  H,  is  the 
quantity  actually  used  to  compute  system  output,  it  is  still  the 
shape  of  H,  the  roll  rate  step  response,  that  is  regarded  as 
the  "motion  signature"  and  independent  variable.  This  is  somewhat 
arbitrary,  of  course.  It  was  chosen  this  way,  however,  because 
intuitively  it  seems  easier  to  think  of  the  control  time  history 
as  a  result  of  the  pilot  making  a  series  of  incremental  steps 
in  control  rather  than  a  series  of  small  discrete  pulses  (although 
there  are  some  marginally  stable  control  situations  that  do  elicit 
a  pulsed  kind  of  control  behavior  from  the  pilot).  This  means  that 
the  independent  variables  in  the  experiment  will  be  described  as 
elements  of  the  step  response,  while  a  sample-data  representation 
of  the  derivative  of  the  step  response  (the  impulse  response)  will 
be  used  in  the  digital  computer  for  the  real-time  computation  of 
system  output. 

An  important  feature  of  the  particular  algorithm  used  is 

the  speed  with  which  the  system  response  is  generated  following 

the  sampling  of  a  new  input.  The  computer  is  triggered  every 

At  seconds  (the  same  At  as  between  the  samples  of  H) .  Between 

pulses,  several  things  must  be  accomplished:  the  current  input 

must  be  sampled;  the  sum  of  all  the  products  of  Fj,  and  must 

be  formed  and  output  as  an  updated  system  response,  G^;  and 

various  bookeeping  tasks  and  buffer  manipulations  must  be  performed. 

Equation  (4a-g)  demonstrates  how,  despite  these  requirements,  a 

very  fast  system  response  is  achieved.  They  are  expansions  of 

Eq.  (3)  at  successive  increments  in  time  for  o  £  t  '•  mAT,  where 
•  •  •  * 

H  is  nAx  long  and  m  >  n.  Remember  that  Hq  ,  .  Hn 

* 

There  are  ways  of  using  an  analog  computer  with  a  digital  computer 
to  simulate  infinite  length  impulse  responses  and  maintain  control 
of  the  shape  of  the  response  on  the  digital  computer.  However, 
this  was  not  necessary  in  the  present  work. 
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Go 

=  F  H  At 
o  o 

(4a) 

G, 

=  F,H  At 

+  {F  H- } At 

(4b) 

1 

1  o 

O  J. 

G2 

=  F  0  H  A  t 

2  O 

+  {F1H1  +  FqH2)At 

(4c) 

Gn 

• 

"  FnHcAT  + 

tFn- A  +  Fn-2F2 

+  .  .  . 

+ 

WAT 

(4d) 

Gn+1 

"  Fn+lVT  + 

^Fn«l  +  Fn-1«2 

+  .  .  . 

+ 

Fl»n)4T 

(4e) 

Gn+2 

=  Fn+2HoAT+ 

{Fn+lHl+Fn  »2 

+  .  .  . 

+ 

F2HnlAT 

(4f  ) 

m 


=  F  H  At 
m  o 


{F  ,11,+  F  0H9  + 
m-1  1  m-2  2 


+  F  H  }At 
m-n  n 


<4g) 


are  stored  in  memory.  Starting  at  t  =  0,  F(t)  is  sampled  to  get 
F  ,  and  the  single  product  in  Eq .  (4a)  is  formed  and  output  as  GQ, 
We  see  that  FQ  will  be  needed  for  products  with  the  other  elements 
of  Hn  to  produce  the  outputs  G  at  subsequent  times.  These  products 
are  formed  now,  stored,  and  Fq  discarded.  Next,  at  t  =  At ,  F(t) 
is  sampled  to  get  F^,  and  the  first  product  in  Eq .  (4b)  is  formed, 
added  to  the  second  product  (which  had  been  formed  at  t  =  o) ,  and 
output  as  G^.  Now  F^  also  will  be  needed  to  form  products  with 
elements  of  Hn  for  future  outputs.  These  are  formed  and  added  to 
the  appropriate  products  already  formed  with  FQ  so  that  the 
quantities  in  brackets  {},  are  being  created  ahead  of  time. 

The  process  continues  in  this  manner  so  that  at  any  time  t  =  ihAt, 
see  Eq .  (4g) ,  the  quantity  in  brackets,  {},  has  already  been 
computed.  When  Fm  is  obtained  by  sampling  F(t),  only  one 
multiply  and  one  add  are  needed  to  produce  the  current  answer, 

Gm.  This  new  result  is  obtained  quickly  and  output  immediately. 
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Then  is  multiplied  by  the  other  Hn  and  these  products 
are  added  to  the  appropriate,  partially  computed  bracketed 
terms  that  are  needed  for  future  outputs. 

The  technique  just  described  results  in  the  overall  sequence 
of  computations  shown  in  the  time-line  of  Figure  1,  which  shows 
the  time  requirements  for  the  major  computational  blocks  of  the 
program  and  when  they  occur  between  interrupts.  The  sample 
time,  that  is,  the  elapsed  time  between  interrupts,  must  be  the 
same  as  the  time  between  the  samples  of  H  stored  in  memory. 

Thus  the  precision  with  which  we  wish  to  represent  H  determines 
At,  and  the  completion  time,  T  ,  is  determined  by  the  number  of 
locations  in  the  H  buffer.  For  the  current  application  we 
chose  Ax  quite  small  (6250  usee)  because  we  are  interested  in 
the  effects  of  some  fairly  small  changes  in  H.  The  H 
buffer  has  128  locations,  which  allow  for  storage  of  impulse 
responses  up  to  0,8  second  long,  which  was  adequate  for  the 
present  work.  Manipulations  of  Ax  and  the  buffer  length  are  of 


sample  time 


=  6250  */SEC 


INTERRUPT 


TOTAL  COMPUTE  TIME 
Tt  =  6100«SEC 


SAMPLE 

Fit) 


OUTPUT 

Gm 


VARIOUS 

CHECKS 


COMPUTE 

H 

m  o 


RESPONSE  TIME 


Tr  -  137  uSEC 


COMPUTE 

Fm”l . Fm% 

completion  time 


Tc  =  5883  uSEC 


100 


t  r  i — p*l — r 

200 


-1 — r 


-V 


300 


TIME,  ajSEC 


[interrupt] 


complete  all 
CALCULATIONS 


WAIT  FOR 
INTERRUPT 


WAIT  TIME 


Tw  =  150«  SEC 


-i — i — i — r 


6100 


6200 


Fig.  1  Time  Line  Between  Interrupts:  Sequence  or  Major  Events  in  the  "Real-Time"  Digital  Computer 
Solution  of  the  Convolution  Integral 
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course  possible  to  accommodate  different  requirements.  Also, 
we  know  that  the  completion  time,  T  ,  could  be  substantially 
reduced  if  necessary,  but  at  the  expense  of  using  more  memory. 

A  very  important  feature  of  the  present  algorithm  is  the  small 
and  constant  response  time  TR  (137  M-sec).  This  is  the  time  between 
sampling  F(t)  and  producing  the  output  G  .  If  the  H 
buffer  were  lengthened,  forcing  the  completion  time,  T  ,  to  get 
longer,  TR  would  not  change.  (Recently,  improvements  have  been 

found  that  will  more  than  halve  the  constant  TD  value.) 

R 

Impulse  responses  made  up  of  straight  line  segments  were 
easily  "drawn"  into  memory  using  a  scheme  that  called  for  typing 
in  the  coordinates  of  each  line  segment  and  specifying  the  time 
between  samples.  (Although  it  was  not  needed  for  the  experiments 
discussed  in  this  report,  a  more  sophisticated  technique  is 
available  for  inputting  curvilinear  functions  when  that  becomes 
necessary.)  Once  drawn,  impulse  responses  can  be  transferred  to 
magnetic  tape  for  easy  recall. 
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3.  EXPERIMENTATION 


APPROACH  TO  THE  PROBLEM 

To  try  out  our  new  approach  to  handling  qualities  research 
employing  the  convolution  simulation  technique  we  chose  to  examine 
aircraft  roll  control  in  single-degree-of-freedom  tracking  tasks. 
This  oft-studied,  idealized  control  task  is  well  suited  to  our 
needs.  Aircraft  roll  response  is  typically  considered  to  be  linear 
and  first  order,  and  this  means  that  there  is  a  single  dynamic 
parameter  (and  a  gain)  that  determines  both  the  shape  of  the 
aircraft  control  response  and  the  flying  qualities.  It  is  the 
aircraft 's  roll-rate  time-constant.  We  argue  that  the  pilot 
does  not  sense  the  time  constant  per  se  and  ask,  "What  is  it 
about  the  shape  of  the  roll  response  with  the  best  time  constant 
that  the  pilot  finds  most  appealing?"  Unfortunately,  we  did  not 
have  the  resources  to  examine  that  question  thoroughly  in  the 
present  effort.  We  were,  however,  able  to  perform  a  few  formal 
experiments  with  naive  subjects  in  an  attempt  to  demonstrate  the 
validity  of  our  approach. 

For  study  purposes  we  have  tentatively  characterized  the 
roll-rate  step-response  by  four  simple  elements,  as  shown  in 
Figure  2.  The  rounding  described  by  E^  is  only  one  of  several 
ways  in  which  a  step  response  can  approach  steady  state.  Some 
preliminary  exploratory  experiments  have  indicated,  however, 
that  subjects  are  in  general  more  sensitive  to  variations  in  the 
initial  part  of  the  response  described  by  the  elements  E^,E2»  and 
E^  than  they  are  to  roundings  or  overshoots  in  the  final  approach 
to  a  steady  state  value.  This,  and  the  fact  of  our  limited 
resources,  led  us  to  restrict  the  first  formal  experiments  to 
variations  in  E^,  E2 ,  E^,  and  Ggs  only,  with  E^  equal  to  zero. 
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Preceding  page  Wank 


TIME 


NOTE  THAT  WHEN  E,  =  Eg  =  Eg  -  E4  =  0,A  PERFECT  RATE  CONTROLLER  IS  SIMULATED 


Fig.  2  Simple,  Four  Element  Parameterization  of  Aircraft  Roll  Motion  in  Response  to  a  Step  Control  Input 


Two  large  experiments  were  performed.  Grand  Experiment  1 

comprises  8  smaller  experiments  (numbered  1.1-1. 8)  in  which 

variations  in  G  and  EQ  are  examined  with  E-,  =  E0  =  0.  Grand 
ss  3  12 

Experiment  2  also  comprises  8  smaller  experiments  (numbered  2.1- 
2.8),  but  with  G  and  E~  equal  to  near-optimum  values  determined 

S  S  j 

in  Grand  Experiment  1  and  E^  and  varied.  Both  experiments 
were  performed  on  a  moving  base  simulator  consisting  of  a  chair 
and  control  stick  assembly  mounted  on  the  motion  platform  of  the 
Research  Hover  Simulator  (RHS)  (Fig.  3).  The  RHS  is  a  six-degree- 
of-freedom  device,  but  for  the  studies  described  here  only  the  roll 
degree-of-freedom  was  active,  roll  motion  being  produced  by 
differential  driving  of  the  two  jacks  supporting  the  motion 
platform  on  either  side  of  the  chair.  Frequency  response  of  the 
jacks  is  "flat"  out  to  4.5  Hz  (Fig.  4).  A  sum  of  sinusoids 
rolling  disturbance  was  introduced,  which  the  subject  attempted 
to  counteract  by  "flying"  the  chair  with  the  control  stick. 

Figure  5  is  a  block  diagram  of  this  set-up.  The  simulator  had 
no  instruments  and  no  special  visual  display.  The  pilot's  visual 
scene  was  a  view  of  one  end  of  the  simulation  laboratory. 

Stick  force  remained  constant  for  both  experiments.  The 
control  stick  was  22  inches  long  and  rotated  in  good  bearings  so 
that  friction  and  breakout  forces  were  not  significant.  Springs 
provided  a  force-feel  of  0.5  pound  per  inch  of  deflection  at  the 
top  of  the  stick. 

Our  choice  of  disturbance  was  somewhat  arbitrary:  There  was 
available  to  us  a  digital  computer  subroutine  that  produced  a 
pseudorandom  signal  as  the  sum  of  ten  independent  sine  waves. 

The  sinusoid  frequencies  were  chosen  to  be  harmonically  unrelated 
and  the  amplitudes  were  informally  adjusted  to  provide  what  was 
judged  to  be  a  representative  disturbance  in  terms  of  pilot 
workload.  The  frequency  and  amplitude  of  the  10  sine  waves  are 
presented  in  Table  1. 
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AMPLITUDE 

RATIO 


Fig.  4  Frequency  Response  of  Moving-Base  Simulatoi 
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SUM  OF 
SINUSOIDS 
DISTURBANCE 


Xg> 


MOVING-BASE 

SIMULATOR 

(PILOT) 


VEHICLE 

DYNAMICS 

(CONVOLUTION) 


G  =  0 


Fig.  5  Block  Diagram  of  Moving-Basa  Experimental  Situation 


TABLE  1  SUM  OF  SINUSOIDS  DISTURBANCE 


Frequency 
(Hz)  ' 

Amplitude  | 

Displacement 

(deg) 

Rate 

(deg/sec) 

0.029 

3. 

78 

0.69 

0.049 

1.17 

0.068 

1.61 

0.107 

2.53 

0.166 

3.93 

0.283 

*  1 

f 

6.70 

0.420 

0.258 

0.69 

0.654 

0.166 

0.69 

1.240 

0.086 

0.69 

2.217 

0.049 

0.69 

Std.  Dev. , 

a 

6.55 

6.04 

PAIR  COMPARISON  TESTING 

To  be  able  to  perform  even  a  small  amount  of  meaningful 
experimentation  within  a  limited  budget,  we  adopted  an  approach 
that  is  unusual  for  modern  flying  qualities  research.  Conventional 
philosophy  would  dictate  the  use  of  a  large  number  of  subjects 
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who  are  both  pilots  and  trained  raters  and  who  would  make  their 
evaluations  using  the  Cooper-Harper  pilot  opinion  scale  (Ref.  1). 
This  is  admittedly  desirable,  but  we  did  not  have  this  kind  of 
subject  readily  available  and  we  could  not  afford  to  hire  any. 

As  a  result  the  authors  served  as  subjects,  although  neither  is 
a  pilot  or  experienced  rater.  This  forced  us  to  abandon  con¬ 
ventional  techniques  in  favor  of  an  experimental  approach  more 
suited  to  the  abilities  of  naive  subjects.  The  most  significant 
problem  is  that  naive  subjects  cannot  draw  on  a  wealth  of 
experience  to  judge  the  merits  of  a  given  configuration  in  some 
absolute  sense.  They  cannot  make  effective  use  of  the  Cooper- 
Harper  scale  because  they  cannot  translate  their  simulator  ex¬ 
periences  into  real  world  experiences.  The  method  adopted 
circumvented  these  difficulties.  The  naive  subjects  made  only 
very  simple  judgments,  but  did  it  many  many  times.  This  method, 
called  pair  comparisons,  requires  only  that  the  subject  be 
able  to  judge  which  of  a  pair  of  configurations  is  the  best. 

The  method  of  pair  comparison  testing  is  based  on  Thurstone'  s 
law  of  comparative  judgement  published  in  1927  (Ref.  2),  and 
is  described  in  the  literature  of  experimental  psychology. 
Guilford' s  text  (Ref.  3)  has  a  lucid  explanation  from  which 
much  of  the  following  description  is  excerpted  and  paraphrased 
to  fit  the  current  application. 

In  the  method  of  pair  comparisons,  all  configurations  of 
interest  are  presented  to  the  simulator  pilot  in  pairs,  and 
typically  all  possible  pairs  are  presented  (although  they 
need  not  be).  In  our  application,  the  pilot  "flies"  each 
configuration  through  a  simple  tracking  task  on  the  simulator: 

30  seconds  with  one  of  a  pair,  followed  immediately  by  30  seconds 
with  the  other.  He  then  judges  which  of  the  pair  he  prefers  — 
that  is,  which  has  better  flying  qualities.  He  must  pick  one 
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even  if  he  feels  it  is  only  a  guess.  Each  pair  is  judged  a 
large  number  of  times,  and  we  have  as  a  numerical  result  the 
number  and  proportion  of  the  times  each  configuration  is  preferred 
over  every  other  configuration.  The  data  are  arranged  in  a  pro¬ 
portion  matrix  such  as  is  shown  in  Table  2.  Each  element  is 
the  proportion  of  the  trials  that  the  configuration  at  the  top 
(identifying  the  column)  was  judged  better  than  the  configuration 
to  the  left  (identifying  the  row).  From  this  data  we  will  produce 
for  each  configuration  a  single  value  on  a  linear  scale  of 
pilot  preference. 


TABLE  2  THE  PROPORTION  MATRIX  SHOWING  THE  PROPORTION  OF 
TRIALS  (Pj>k)  THAT  EACH  CONFIGURATION  (Cj )  IS 
PREFERRED  OVER  EVERY  OTHER  CONFIGURATION  (Cfc) 


C 

a 

Cb 

C 

c 

C. 

J 

*  *  Cn 

Ca 

Pa>a 

Pb>a 

Pc>a 

’  Pj>a  * 

Pn>a 

Cb 

Pa>b 

Pb>b 

Pc>b 

Pj>b 

Pn>b 

C 

c 

Pa>c 

Pb>c 

P  . 
c>c 

P.. 

J>c 

Pn>c 

• 

Ck 

Pa>k 

Pb>k 

•  • 

Pc>k 

•  •  •  • 

.  .  p 

j>k 

•  •  • 

Pn>k 

Cn 

P  . 
a>n 

Pb>n 

Pc>n 

•  •  •  • 

P.v 

j>n 

•  •  • 

•  •  P  . 
n>n 

n 

n 

n 

n 

n 

1 

^L?a>k 

Sb>k 

2pok  • 

'  '  Ipj>k- 

^Pn>k 

k=a 

k=a 

k«a 

k*a 

k-a 

Note:  The  Cn’ s  are  always  ordered  so  that  the  sums  at  the 

bottom  increase  (or  decrease)  monotonically  from  left 
to  right. 
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The  casual  reader  can  skip  the  following  details  of  pair 
comparison  analysis  and  resume  with  the  last  paragraph  on  page  24. 

The  transformation  of  comparative  judgement  data  into  scale 

l 

values  starts  with  Thurstone' s  law  of  comparative  judgement 
(Ref.  2).  It  is  stated  as  follows: 


ARjk  Rj  -  Rk  -  zjk 


V»j 


2  +  °k2-2rjk  aj°k 


(5) 


where 


R^  and  R^ 


mean  pilot  impression  of  the  flying 
qualities  of  configurations  C^.  and  C^, 


ZM 

jR 


a.  and  a, 
J  R 


respective ly 

■  standard-measure  distance  or  deviate  from 
the  mean  of  a  unit  normal  distribution 

»  standard  deviations  of  the  distributions 
of  R^  and  R^k»  respectively 

rjk  "  coefficient  of  correlation  between  R^ 

and  \k- 

The  radical  term  is  the  standard  deviation  of  the  differences 

R^j  “  R^k  and  is  the  unit  of  the  scale  on  which  each  separation 

of  pilot  preference  (R^  -  R^)  is  expressed.  The  size  of  each 

separation  R ^  -  R^  can  be  determined  and  a  scale  of  pilot 

preference  developed  if  we  know  the  values  on  the  right-hand 

side  of  Eq.  (5).  We  can  find  Z^k  from  the  experimentally  determine 

proportions  of  P.  ,  ,  but  the  remaining  parameters  are  unknown.  We 
j  >k 

must,  therefore,  make  some  simplifying  assumptions.  We  assune 
that  the  correlation  coefficients  are  all  the  same  and  that 
the  standard  deviations  are  all  the  same.  Equations  (5)  then 
becomes 


ARjk  "  Rj  "  ^  ”  Zjk 


V2(l-r) 


(6) 
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The  term  in  brackets  is  now  a  constant.  It  can  be  thought  of 
as  the  unit  of  the  scale  of  pilot  preference  and  therefore  set 
equal  to  unity. 


Thus , 


(7) 


becomes  the  final  simplified  expression. 

There  is  of  course  a  risk  involved  in  making  the  above 
assumptions.  The  risk  is  minimized,  however,  by  the  fact  that 
after  we  have  scaled  a  set  of  configurations  according  to 
pilot  preference  using  Eq.  (7),  we  can  check  the  results  to  see 
whether  they  are  internally  consistent.  The  test  of  internal 
consistency  was  developed  by  Mosteller  (Ref.  4),  and  involves 
going  backwards  from  the  scale  values  calculated  to  produce  a 
new  proportion  matrix.  This  would  be  the  one  expected  given  the 
scale  values  calculated.  A  chi-square  test  is  then  applied  to 
determine  the  goodness  of  fit  of  the  expected  proportions  to  the 
original  experimentally  obtained  proportions.  If  the  results 
are  internally  consistent,  we  may  say  that  we  have  found  nothing 
to  contradict  the  assumptions  made.  The  test  of  internal 
consistency  is  elucidated  in  the  appendix  where  it  is  applied 
to  the  data  of  this  study. 


To  create  a  scale  from  the  data  we  next  use  normal  curve 

tables  and  Eq.  (7)  to  transform  the  proportion  matrix  into  a  AR^ 

matrix  like  the  one  shown  in  Table  3.  Each  AR.,  element  is  an 

J 

estimate  of  the  distance  R^  -  R^,  where  the  latter  are  the  as 
yet  unknown  positions  of  configurations  and  on  the  scale  of 
pilot  preference.  To  get  the  best  estimates  of  the  distances 
between  configurations  we  must  utilize  all  the  data.  For  example, 
AR^  is  but  one  estimate  of  the  scale  distance  from  configuration 
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TABLE  3  THE  AR^k  MATRIX  SHOWING  ESTIMATES  OF  SEPARATIONS 
(ARjk)  BETWEEN  EACH  CONFIGURATION  (C^ )  AND  EVERY 
OTHER  CONFIGURATION  (Ck>  ON  A  SCALE  OF  PILOT 
PREFERENCE 


Ca 

Cb 

c 

c 

• 

C. 

J 

.  .  .  C^ 

Ca 

AR 

aa 

ARba 

AR  a 

ca 

• 

•  ARja 

.  .  .  AR 

na 

Cb 

AR  , 
ab 

ARbb 

AR  , 
cb 

• 

•  ARjb 

.  .  .  AR  . 

nb 

C 

c 

AR 

ac 

ARbc 

AR 

cc 

• 

• 

•  aRjc 

.  .  .  AR 

nc 

Ck 

ARak 

ARbk 

ARck  • 

• 

•  ARjk 

.  .  .  AR  . 

nk 

Cn 

AR 

an 

ARbn 

AR 

cn 

AR. 

jn 

.  .  .  AR 

nn 

n 

n 

n 

n 

n 

Sum  of 
Columns 

2ARak 

IARbk 

SARck  • 

• 

•  2av 

•  •  •  2iRnk 

k*a 

k*a 

k«a 

k—a 

k—a 

Mean 

(Scale 

Value) 

Ma 

“b 

M 

c 

• 

M. 

3 

.  .  .  M 

n 

Ca  to  configuration  C^  based  on  a  single  proportion  representing 
direct  comparisons  of  Ca  and  C^.  These  two  configurations  were 
also  compared  with  all  other  configurations,  and  their  relative 
preferences  when  paired  with  others  are  useful  additional  data. 

We  actually  have  n  estimates  of  any  (R^  -  Rk)  as  shown  in  Eq.  (8) 
for  (Ra  -  Rb). 
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ARaa  -  ARba  '  Ra  '  Ra  '  "b  +  Ra  *  <Ra  '  V  (8a) 

ARab  -  ARbb  "  Ra  •  "b  '  \  +  \  -  <Ra  '  Kb)  (8b) 

ARac  -  ARbc  -  Ra  -  Rc  -  Kb  +  Rc  -  <Ra  '  V  (8c) 


ARan  -  ARb„  ■  Ra  '  Rn  '  Kb  +  Rn  -  <Ra  '  Kb>  (8n) 

The  mean  of  the  estimates  can  be  obtained  by  averaging  the  column 
of  (R^  -  R^)  at  the  far  right.  Note  that  the  two  columns  of  AR^ 
at  the  far  left  of  Eq  (8)  are  identical  with  the  first  two 
columns  of  the  AR.,  matrix  in  Table  3.  Since  the  difference 
between  the  means  is  equal  to  the  mean  of  the  differences,  we 
can  arrive  at  the  same  end  result  by  summing  the  columns  first 
and  then  finding  the  two  means.  We  can  do  this  for  all  the 
columns  in  the  AR^  matrix,  and  since  the  differences  between 
neighboring  pairs  of  configurations  are  mean  estimates  of  their 
appropriate  separations,  the  means  themselves  will  serve  as 
scale  values. 

Thus  we  arrive  at  a  scale  of  pilot  preferences  for  the  con¬ 
figuration  compared  one  to  another.  Pair  comparison  testing 
is  not  without  its  drawbacks.  The  principal  one  arises  from 
the  fact  that  it  provides  no  information  about  a  configuration  in 
any  absolute  sense,  but  only  in  terms  of  the  differences  between 
configurations.  The  scale  produced  is  an  interval  or  equal-unit 
scale  where  equal  numerical  distances  represent  equally  distant 
pilot  preferences,  but  where  the  zero  point  is  arbitrary.  In 
certain  instances  this  could  make  it  difficult  to  compare  the 
results  of  different  pair  comparison  tests.  Another  criticism 
is  that  repeating  the  same  tracking  task  over  and  over  to  create 
all  the  elements  of  the  proportion  matrix  is  a  boring  task  in 
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which  the  subjects  must  strive  to  remain  alert.  Despite  these 
drawbacks,  pair  comparison  testing  proved  to  be  satisfactory  for 
this  preliminary  work,  and  it  allowed  us  to  make  effective  use 
of  the  experimenters  as  subjects. 

EXPERIMENTAL  DESIGN  AND  PROCEDURE 

To  demonstrate  the  utility  of  the  convolution  algorithm  as 
a  means  for  manipulating  aircraft  control  responses  in  handling 
qualities  research,  two  large  experiments  were  performed:  Grand 
Experiment  1  and  Grand  Experiment  2.  Each  comprised  eight 
small  pair  comparison  experiments  (Experiments  1.1,  1.2,  ...,  1.8 
and  Experiments  2.1,  2.2,  ...,  2.8)  in  which  a  single  element 
of  aircraft  roll  response  was  varied.  The  experimental  spaces 
for  the  two  large  experiments  are  shown  in  Figures  6  and  7,  and 
within  each  space  the  various  pair  comparison  experiments  are 
indicated.  Grand  Experiment  1  was  designed  as  a  first  exploratory 
look  at  some  details  (G  and  E,)  of  a  response  that  is  generally 
first-order  in  character  and  with  no  control  system  anomalies 
(such  as  time  delay).  Conversely,  Grand  Experiment  2  was  designed 
as  a  first  exploratory  look  at  the  relative  effects  of  two  kinds 
of  control  system  anomalies  (E^  and  E2)  when  the  general  first- 
order  like  response  is  near  the  optimum  one  determined  from 
Grand  Experiment  1.  Table  4  is  a  summary  of  the  50  configurations 
that  were  examined  in  the  two  grand  experiments. 

Figures  6  and  7  reveal  an  important  strategy  used  to  reduce 
the  effective  size  of  the  grand  experiments;  that  is,  that  we 
do  not  compare  each  configuration  with  every  other  configuration 
in  the  experimental  space.  Rather,  each  configuration  is  only 
compared  with  a  small  subset  of  configurations  in  either  one  or 
two  very  small  pair  comparison  experiments  (very  small  relative 
to  the  case  where  each  configuration  is  compared  with  all  others). 
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STEADY  STATE  ROLL  RATE  GAIN,  Ggg,  DEG/SEC/IN. 

Note:  Pair  comparison  experiments  are  indicated  by  lines  connecting  the  circled  configurations  studied,  and 
the  experiment  number  is  called  out  in  a  box  at  one  end  of  the  line.  For  instance,  experiment  1.3 
studied  Ggs  values  of  6,  9,  12,  18  and  24  deg/sec/in.  with  E3  =  0.4  sec.  The  C's  in  the  circles  are  the 
configuretion  numbers  as  described  in  Table  4. 


Fig.  6  Experimental  Space  for  Grand  Experiment  1 
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Fig.  7  Experimental  Space  for  Grand  Experiment  2 


TABLE  4  STEP  RESPONSE  CONFIGURATIONS  STUDIED 


TABLE  4  STEP  RESPONSE  CONFIGURATIONS 


Configurations 

Element 

E1 

(sec) 

E2 

(sec) 

E3 

(se 

C3l 

tr 

0 

0; 

c32 

0.04 

C33 

0.08 

C34 

0.12 

C35 

1 

r 

0.16 

C36 

08 

0 

C37 

0.04 

C38 

0.08 

C39 

0.12 

C40 

1 

r 

0.16 

C41 

yj 

12 

0 

C42 

0.04 

C43 

0.08 

C44 

0.12 

C45 

1 

t 

0.16 

C46 

■ 

16 

0 

C47 

~9 

20 

0 

C48 

16 

0.04 

C49 

0. 

16 

0.08 

C50 

0. 

16 

0.12 

For  instance,  Figure  7  shows  that  is  compared  with  C2Q,  c^9’ 
and  C^q  in  Experiment  2.4  and  compared  with  C^,  C22’  ^33 
and  C35  in  Experiment  2.6,  but  that  it  is  not  compared  with  any 
of  the  other  17  configurations  in  the  experimental  space. 

Obviously  this  provides  for  substantial  savings  in  experimentation. 
It  means,  however,  that  we  cannot  directly  place  all  the  con¬ 
figurations  in  a  Grand  Experiment  on  a  single  scale  of  pilot 
preference.  Instead,  pilot  preference  scales  are  first  obtained 
for  each  of  the  smaller  experiments  (1.1  through  1.8  or  2.1 
through  2.8)  and  then  these  are  adjusted  to  minimize  the 
difference  between  scale  values  obtained  for  the  same  configuration 
in  two  different  experiments  (i.e.,  where  two  scales  cross). 

The  adjustments  come  in  the  form  of  constants  that  are  added 
to  the  scales,  and  the  set  of  eight  constants  needed  to  adjust 
the  eight  scales  obtained  in  a  grand  experiment  were  chosen  to 
minimize,  in  a  least  squares  sense,  the  sum  of  all  15  (16  in 
Grand  Experiment  2)  differences  in  pilot  preference  that  occur 
wherever  two  scales  cross.  This  maneuver  is  valid  for  two 


reasons.  First,  in  simplifying  Thurstone' s  Law  from  Eq.  (5) 
to  Eq.  (7),  we  have  already  made  the  assumption  that  all  the 
scale  separation  elements  (AR^)  in  a  grand  experiment  have 
the  same  units ,  and  the  individual  pair  comparison  experiments 
are  viewed  as  the  way  to  estimate  the  differences  between  selected 
subsets  of  configurations.  Second,  the  pair  comparison  tests, 
as  noted  earlier,  produce  interval  scales  of  pilot  preference 
where  the  distance  between  configurations  is  prescribed, 
but  where  the  whole  scale  can  be  arbitrarily  shifted  up  or  down 
by  a  constant  amount.  One  thing  is  lost,  however,  and  that 
is  the  ability  to  apply  the  test  of  internal  consistency  over 
a  whole  grand  experiment.  Only  the  individual  one -dimensional 
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scales  produced  by  the  separate  smaller  pair  comparison  experi¬ 
ments  can  be  checked  for  violations  of  the  assumptions  made  in 
simplifying  Thurstone' s  Law. 

A  significant  part  of  our  experimental  design  effort  was 
spent  on  informal  experimentation  to  determine  the  levels  and 
ranges  of  variation  of  the  independent  variables.  This  has  to 
be  done  very  carefully  when  preparing  to  do  pair  comparison 
testing.  Determining  the  proper  range  of  variation  is  very 
important  because  any  two  configurations  in  an  experiment  must 
never  be  so  "far  apart"  that  one  would  always  be  judged  better 
than  the  other.  This  would  put  ones  and  zeroes  in  the  proportion 
matrix,  and  these  extreme  values  cannot  be  used  in  the  scaling 
process.  Actually,  it  is  common  practice  to  consider  all 
proportions  greater  than  0.977  (and  less  than  0.023)  too  extreme 
and  omit  them  from  the  scale  calculations  (Ref.  3). 

The  number  of  levels  of  the  independent  variables  is 
important  because  it  determines  the  number  of  configurations 
(C)  under  study,  and  the  number  of  pairs  (P)  increases  quad- 
ratically  with  the  number  of  configurations. 

P  -  \  C  (C  -  1)  (9) 

Consider  also  that  an  inherent  part  of  pair  comparison  testing 
is  repetitive  evaluation  of  each  pair.  The  total  number  of 
trials  (T)  then  becomes: 

T  -  \  N  C  (C  -  1)  (10) 

where:  C  is  the  number  of  configurations 

N  is  the  total  number  of  times  each  pair  of 
configurations  is  evaluated  (independent  of 
order  of  presentation) 
and  T  is  the  total  number  of  trials 
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In  our  work  we  had  two  subjects,  and  for  most  of  the  16  pair 
comparison  experiments  each  subject  evaluated  each  pair  10 
times  so  that  N  was  20  (for  experiments  1.7,  2.3  and  2.7  the 
numbers  were  doubled,  so  that  N  was  40).  Figure  8  is  a  plot  of 
Eq.  (10)  for  N  equal  to  20.  It  demonstrates  the  need  to  keep 
the  number  of  configurations  to  a  minimum.  An  increase  of 
from  5  to  6 ,  for  instance,  increases  the  total  number  of  trials 
required  by  100. 

For  most  of  the  variables  considered  a  compromise  had  to  be 
reached  to  achieve  the  minimum  number  of  configurations  that 
could  be  used  to  cover  the  desired  range  of  variation.  For 
instance  consider  the  levels  of  used  in  experiments  1.6,  1.7, 
and  1.8.  Here,  informal  experimentation  had  indicated  that  we 
should  examine  E^' s  as  large  as  0.6  second.  To  do  this  and  keep 
the  increment  between  configurations  reasonably  small  (0.1  second) 
meant  having  as  many  as  seven  configurations  if  we  were  to 
examine  all  the  smaller  E^' s  down  to  E^  =  0.  Despite  the  desire 
to  include  the  very  small  values  of  E^,  economy  dictated  their 
elimination.  Going  from  7  to  5  configurations  cut  the  number 
of  trials  by  more  than  half:  from  420  to  200  (see  Fig.  8). 

We  also  reasoned  that  very  small  values  of  E^  are  hard  to  obtain 
in  the  real  world,  particularly  in  the  highly  augmented  aircraft 
to  which  this  work  is  addressed.  Thus  we  settled  on  the  five 
E^  values  used. 

The  selection  of  values  for  E^  and  E2  used  in  Grand  Experiment 
2  (see  Fig.  7)  were  also  strongly  affected  by  the  results  of 
informal  experiments .  They  showed  that  the  limits  of  control- 
ability  were  being  approached  when  the  sum  E^  +  E2  reached  0.25 
second,  and  that  even  relatively  small  values  of  either  E^  or 
E2  were  noticeably  detrimental.  The  latter  result  led  us  to 
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choose  0.04  second  as  the  increment  for  both  E^  and  E2*  To 
limit  the  total  number  of  trials,  however,  the  range  of  variation 
had  to  be  slightly  compromised.  E^  and  E2  did  not  go  above 
0.16  second  except  in  experiments  2.1  and  2.5,  where  values 
up  to  0.20  second  were  allowed.  Even  so  there  were  several 
configurations  where  E^  +  E2  equaled  or  exceeded  0.24  second 

(c40’  C44’  C45 ’  C49  and  C50^‘ 

Tables  5  and  6  present  the  experiment  matrices  for  Grand 
Experiments  1  and  2,  respectively.  Elements  in  the  matrix 
represent  pairs  of  configurations,  and  the  set  of  pairs  on  one 
side  of  a  diagonal  is  a  mirror  image  of  the  set  of  pairs  on  the 
other  side.  Each  pair  on  one  side  of  the  diagonal  contains  the 
same  configurations  as  its  mirror  image  on  the  other  side,  but 
the  order  of  presentation  is  reversed.  Biasing  due  to  order 
of  presentation  was  counteracted  by  doing  the  balanced  design; 
that  is,  by  evaluating  the  pairs  on  both  sides  of  the  diagonal 
an  equal  number  of  times  .  We  did  not  evaluate  the  pairs  on 
the  diagonal.  They  represent  identical  configurations  and 
theoretically  produce  proportions  of  0.5.  The  extent  to  which 
they  do  not  is  useful  only  to  correct  biased  data  from  unbalanced 
designs . 

Note  that  one  or  more  pairs  of  configurations  have  been 
deleted  from  the  upper  right  and  lower  left  hand  corners  of  the 
experimental  matrices  for  experiments  2.1  through  2.8  (Table  6). 


In  Eq.  (9),  the  number  of  pairs,  P,  referred  to  is  the  number 
of  completely  dissimilar  pairs.  That  is,  the  number  of  pairs  on 

one  side  of  the  diagonal.. _ The  total  trials,  T,  in  Eq.  (10), 

however,  include  all  trials  independent  of  order  of  presentation. 
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TABLE  5  EXPERIMENT  MATRICES:  PAIRS  OF  CONFIGURATIONS*  (C.,C,)  FOR 
GRAND  EXPERIMENT  1  J 


rhe  C.  are  the  configurations  presented  first  in  each  pair,  and  tne  C^  are  presented 
last.**  All  configurations  are  defined  in  Table  4. 


TABLE  6  EXPERIMENT  MATRICES:  PAIRS  OF  CONFIGURATIONS* 
(C.,  C.  )  FOR  GRAND  EXPERIMENT  2 


Experiment  2.3  d)  Experiment  2.4 

C.  are  the  configurations  presented  first  in  each  pair,  and  tne  C^  are  presented 
. ^  All  configurations  are  defined  in  Table  4. 


These  economies  stemmed  from  informal  experiments,  which 
suggested  that  increases  in  or  E2  would  always  be  detrimental. 
Thus  we  could  make  an  educated  guess  about  what  the  rank  order 
of  the  configurations  would  be  in  the  proportion  matrices  before 
the  experiments  were  performed.  It  was  assumed  they  would  be 
ordered  according  to  decreasing  values  of  the  variable  (either 
Ei  or  E2),  and  that  this  would  correspond  to  a  monotonic  increase 
in  pilot  preference.  Thus  it  was  assumed  that  each  proportion 
matrix  in  Grand  Experiment  2  would  end  up  being  ordered  the 
same  way  as  its  respective  experiment  matrix.  This  assumption 
(which  the  experiments  ultimately  justified)  made  it  possible 
to  select  the  pairs  of  configuration  that  could  be  logically 
omitted  from  the  experiments  for  the  sake  of  economy.  The  pairs 
in  the  corners  of  the  proportion  matrices  furthest  from  the 
diagonal  are  the  ones  most  likely  to  produce  proportions  close 
to  zero  (or  one)  and  are  the  most  risky  for  use  in  producing 
scale  values.  Thus  28  pairs  (15  percent  of  the  total)  were 
deleted  from  the  experiment  matrices  for  the  eight  experiments 
comprising  Grand  Experiment  2. 

Despite  this  strong  effort  to  economize.  Grand  Experiments 
1  and  2  still  remained  large.  A  total  of  3680  trials  were 
required  to  complete  them:  1800  for  Grand  Experiment  1  and  1880 
for  Grand  Experiment  2.  Even  so,  the  experimentation  went  fairly 
rapidly.  The  3680  trials  were  completed  in  about  30  days  of 
running,  over  a  period  of  several  months.  Because  some  of  our 
laboratory  facilities  were  shared  with  other  programs,  we  would 
typically  run  for  periods  of  3  to  5  days.  In  this  work,  one 
subject  served  as  the  "pilot"  while  the  other  acted  as  the 
"experimenter,"  swapping  roles  after  every  10  trials.  Each 
kept  the  records  of  the  other  so  that  neither  could  gain  knowledge 


of  his  own  performance  prior  to  the  completion  of  an  experiment. 
The  experimenter  working  at  the  console,  would  call  up  from 
computer  memory  a  pair  of  configurations.  (The  computer  used 
was  an  Ambilog  200  Stored  Program  Signal  Processor  made  by 
Adage,  Inc.,  Boston,  Mass.)  He  would  start  the  problem,  and 
the  pilot  would  begin  tracking  with  the  first  configuration. 

After  30  seconds  the  disturbance  would  stop,  and  a  light  would 
start  flashing  at  the  pilot' s  station,  signalling  the  end  of 
tracking  with  the  first  configuration.  The  flashing  continued 
for  4  seconds,  after  which  tracking  with  the  second  configuration 
began.  At  this  time  the  light  stopped  flashing  but  remained  lit 
to  remind  the  pilot  that  he  was  tracking  with  the  second 
configuration  of  a  pair.  After  another  30  seconds  the  disturbance 
stopped  again  and  the  light  went  off,  signalling  the  end  of  the 
trial.  The  pilot  indicated  which  configuration  he  preferred 
by  pushing  either  of  two  buttons  on  a  control  box  mounted  on  his 
seat.  His  response  was  displayed  on  a  cathode  ray  tube  at  the 
computer  console  and  was  recorded  by  the  experimenter  who  then 
started  the  cycle  again  by  loading  the  next,  randomly  selected 
pair  of  configurations.  In  general,  all  the  pairs  for  all  the 
comparison  experiments  composing  each  grand  experiment  were  put 
together  into  two  large  random  sequences  of  trials.  Excluded 
from  these  two  lists  were  the  pairs  for  experiments  1.7,  2.3, 
and  2.7.  They  were  run  as  three  separate  experiments  before 
the  large  groups  of  trials  as  a  final  check  on  the  levels  and 
ranges  of  variation  of  the  independent  variables.  They  were 
also  used  to  verify  that  an  N  of  20  was  adequate. 
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4.  DISCUSSION  OF  RESULTS 


The  details  of  the  data  analysis  for  the  experiments  described 
in  the  previous  section  are  reported  in  the  appendix.  Briefly, 
for  each  grand  experiment  a  cubic  polynomial  in  the  two 
experimental  variables  was  fit  in  a  least  squares  sense  to  the 
pilot  preference  data,  and  the  graphs  presented  here  are  mostly 
planar  sections  of  these  cubic  surfaces.  Although  we  discuss  the 
impact  of  these  results  on  understanding  the  pilot' s  flying 
qualities  preferences,  we  stress  the  larger  purpose  of  the  study; 
to  assess  the  feasibility  of  using  real  time  convolution  on  a 
digital  computer  to  simulate  system  dynamics  for  studying  the 
flying  qualities  of  highly  augmented  craft. 

The  results  of  Grand  Experiment  1  are  summarized  in  Figures 

9  and  10.  Figure  9  contains  plots  of  pilot  preference  versus 

steady  state  gain,  Ge.  at  Eo  values  of  0.2,  0.3,  0.4,  0.5  and 

0.6  second  that  were  determined  from  experiments  1.1,  1.2,  1.3, 

1.4  and  1.5,  respectively.  The  pilot  preference  scale  on  the 

ordinate  ranges  from  the  "worst"  or  least  preferred  configuration 

to  the  "best"  or  most  preferred  configuration  of  Grand  Experiment 

1.  Values  in  between  are  regarded  as  being  some  percentage 

of  the  best;  that  is,  a  configuration  with  a  pilot  preference 

rating  of  75  percent  is  said  to  be  regarded  by  the  pilots  as  being 

75  percent  as  good  as  the  best  configuration  examined.  Figure  9 

shows  a  well  defined  optimum  steady  state  gain,  G  ,  that  only 

s  s 

varies  between  the  values  of  12  and  15  deg/sec/in.  over  the  whole 

range  of  E^  values  (0.2  sec  £  E-j  £  0.6  sec).  This  small  range 

for  the  optimum  G  strongly  suggests  that  it  may  indeed  be 

s  s 

some  sort  of  a  fundamental  parameter  as  far  as  the  pilot  is 
concerned. 
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Figure  10  shows  that  there  is  also  a  well  defined  maximum 

"rise  time,"  Eg,  and  that  the  fastest  response  (i.e.,  when  Eg  =  0) 

is  never  the  best.  This  is  important  because  although  modern 

control  system  technology  might  contrive  a  way  to  provide  very 

fast  responses  it  would  no  doubt  come  at  the  expense  of  added 

mechanical  complexity  or  exorbitant  control  power  requirements, 

or  both.  The  two  curves  for  G  equal  to  9  and  18  deg/sec /in. 

s  s 

cover  the  range  of  optimum  G  values  (12  to  15  deg/sec/in.) 

s  s 

determined  from  Figure  9,  and  from  them  we  see  that  the  optimum 
E^  value  must  fall  within  0.225  to  0.375  second. 

Although  the  optimum  G  remains  fairly  constant  with 

s  s 

increasing  Eg  (see  Fig.  9),  we  see  that  at  the  highest  values 
of  E,  (0.5  and  0.6  second)  the  pilot  preference  versus  G 

*5  S  S 

curves  are  becoming  noticeably  flatter.  If  this  were  the 
precursor  to  the  optimum  G  value  increasing  much  more  rapidly 
with  further  increases  in  Eg,  it  might  indicate  that  there  is 
a  minimum  initial  acceleration  (G  /Eo)  wanted  by  the  pilot 
independent  of  G  and  E,  alone.  This  conjecture  is  belied, 
however,  by  Figure  11  where  we  have  plotted  pilot  preference 
versus  G  /E,.  The  minimum  acceptable  "acceleration"  for 

S  S  j 

a  given  pilot  preference  is  clearly  not  unique. 

Another  interesting  way  to  look  at  the  results  of  Figures 
9  and  10  is  to  convert  them  into  the  bounded  regions  of 
Figure  12.  These  delineate  where  the  "knee"  of  the  idealized 
roll-rate  step-response  curve  must  fall  if  the  response  is  to 
achieve  the  specified  level  of  pilot  preference  (e.g. ,  the 
90  percent  region  contains  the  "knee"  of  all  responses  that  are 
at  least  90  percent  as  good  as  the  "best"  response).  It  should 
be  noted  that  the  experimenter  felt  that  only  the  very  best 
of  the  configurations  were  actually  good.  Thus  we  might 
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"INITIAL  ACCELERATION",  Ggg/Eg,  DEG/SEC2/IN. 

Fig.  11  Pilot  Preference  versus  "Initial  Acceleration' 


CORNERS  (4)  OF  EXPERIMENTAL  SPACE 


"RISE  TIME".  Eg  OR  TIME,  t,  SEC 


Fig.  12  Most  Preferred  Regions  for  the  "Knee"  of  the  Idealized  Roll  Rate  Step  Response 


conclude  that  the  good  configurations  are  found  only  in  the  95 

or  98  percent  regions.  This  presentation  is  really  the  most 

graphic,  and  it  shows  that  for  the  idealized  control  systems 

being  considered  the  best  configurations  (i.e.,  those  most 

preferred  by  the  pilots)  have  step-responses  that  lie  in  a 

small  well  defined  region  of  the  step-response  plane.  Note 

that  in  this  figure  Eg  equals  t  (for  0  £  t  £  £3)*  and  Gss 

equals  4>  (for  Eg  £  t  £  00) .  For  the  95  percent  region  Eg 

varies  at  most  between  0.17  and  0.37  second  and  G  varies  at 

ss 

most  between  about  10  and  15  deg/sec/in.  This  is  of  course 
consistent  with  the  narrow  range  of  optimum  values  for  both  G 

s  s 

and  Eg  displayed  by  Figures  9  and  10.  Also  shown  are  a  line  of 

optimum  E,  as  a  function  of  G  and  a  line  of  optimum  G  as  a 
j  s  s  s  s 

function  of  E«.  The  optimum  G  line  is  very  flat  and  demonstrates 

<3  S  S 

quite  clearly  that  the  best  value  for  G  is  virtually  independent 

s  s 

of  Eo ;  once  again  implying  that  G  is  in  fact  a  fundamental 

j  SS 

parameter  that  the  pilot  is  sensitive  to  in  his  evaluation  of 
flying  qualities. 

Next  we  would  like  to  validate  our  data  by  comparing  it 
with  some  flight  test  experiments  performed  by  Princeton  University 
(Ref.  5).  For  purpose  of  comparison  we  define  in  Figure  13  how 
the  idealized  roll  rate  response  used  here  is  approximated  by 
the  linear  first  order  response  used  in  Ref.  5  (or  vice  versa). 

The  Princeton  work  cited  here  examined  roll  control  sensitivity 
and  rolling  time  constant  in  a  variable-stability  airplane  that 
was  flown  through  a  landing  approach.  Flights  were  made  in  calm 
air  with  a  "moderate"  level  of  turbulence,  "such  as  might  be 
present  near  a  squall  line,"  imposed  on  the  vehicle  by 
manipulation  of  control  surfaces  through  the  autopilot.  Thus 
the  flight  task  was  quite  similar  to  the  one  used  in  our  simulator 
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ROLL  RATE,  $ 


TIME,  t 

Fig.  13  Comparison  of  First  Ordar  Linaar  Roll  Rata  Rasponsa  and  tha  Grumman  Idoalizad  Roll  Rata  Rasponsa 
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studies.  Results  of  both  studies  are  shown  in  Figure  14.  The 
Grumman  data  is  a  transformation  of  the  95  percent  region  of 
Figure  12  onto  the  plane  of  Figure  14,  while  the  Princeton 
data  is  a  reproduction  of  the  way  it  was  presented  in  Ref.  5. 
Although  the  Princeton  results  show  a  slightly  wider  range  of 
satisfactory  t' s  than  the  comparable  range  of  E^'s  for  the 
95  percent  region,  the  results  are  really  quite  similar  — 
especially  in  light  of  our  uncertainty  about  what  level  of 
pilot  preference  (95  percent,  90  percent,  85  percent,...)  is 
comparable  to  a  "satisfactory"  rating  of  3  on  the  Cooper  Scale 
(Ref.  1).  A  slightly  lower  level  of  pilot  preference  would 
have  a  range  of  "rise- times"  almost  identical  to  Princeton's 
satisfactory  region. 

The  principal  discrepancy  between  the  two  sets  of  data  is 
the  difference  in  the  general  level  of  "initial  acceleration" 
found  to  be  satisfactory.  Here  we  feel  that  this  difference 
can  be  largely  explained  by  the  one  major  difference  between 
the  two  experimental  set-ups:  stick  force  gradient.  This 
parameter  was  not  varied  in  either  study;  consequently,  we  could 
not  assess  its  direct  effect  on  these  data.  The  stick  force 
gradient  is  generally  known  to  have  a  powerful  effect  on  flying 
qualities  in  tracking  tasks  (see,  for  instance.  Ref.  6),  and 
there  was  a  big  difference  between  the  values  used  in  the  two 
studies.  For  the  Princeton  study  the  stick  force  gradient  was 
4.5  pounds/inch  (representative  of  contemporary  jet  fighter 
aircraft),  while  for  our  work  it  was  only  0.5  pound/inch 
(more  representative  of  hovering  VTOL  craft).  The  much  higher 
value  used  in  the  Princeton  study  would  make  the  pilots  want  a 
higher  "initial  acceleration"  gain  to  reduce  the  physical  effort 
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Fig.  14  Comparison  of  Grumman  Data  with  Prince ton  Univarsity  Data  (Ref.  5) 
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required  to  manipulate  the  controller.  Conversely,  the  much 
lower  value  used  in  our  study  would  make  the  pilots  want  a  lower 
"initial  acceleration"  gain  to  reduce  any  tendency  toward  over¬ 
control  and  pilot-induced-oscillation  (PIO).  With  this 
explanation  in  mind,  and  remembering  that  the  step  responses  are 
not  exactly  alike  in  the  two  studies,  we  conclude  that  our  results 
are  wholly  reasonable  in  comparison  to  the  Princeton  flight 
test  data.  The  Princeton  report  also  showed  that  the  optimum 
configurations  could  be  expressed  in  terms  of  a  given  bank 
angle  attained  in  one  second  for  one  inch  stick  deflection.  It 
turns  out  that  the  optimum  region  is  about  the  same  for  all 
roll  time-constants  and  corresponds  to  a  24  degree  bank  angle 
in  one  second  (see  Fig.  14)  for  the  control  input  shown  below. 


Time,  t,  sec 

For  comparison,  we  tried  to  develop  something  similar  for 
our  95  percent  region  using  the  same  control  input.  Figure  14 
shows  that  a  line  of  9.5  degrees  in  one  second  for  one  inch 
control  deflection  closely  matches  our  optimum  region.  This 
hints  at  the  possibility  that  pilots  regard  their  ability  to 
achieve  a  given  bank  angle  within  one  second  for  a  given  control 
effort  as  a  fundamental  measure  of  a  system’ s  worth  in  a  roll 
tracking  task.  We  saw  in  Figures  9  and  12,  however,  that  steady 
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state  gain,  G  ,  might  be  a  fundamental  parameter  that  pilots 

choose  independently  of  E^.  Thus  our  data  suggests  a  hypothesis 

that  says,  "the  pilot  always  chooses  the  same  steady  state  gain, 

G  ,  (for  a  given  stick  force  gradient)  independent  of  the  tran- 

sient  character  of  the  response,"  but  when  viewed  differently  it 

also  suggests,  along  with  the  Princeton  data,  a  hypothesis  that 

says,  "the  pilot  chooses,  G  ,  such  that,  with  a  certain  step- 

s  s 

like  command  of  one  inch,  the  aircraft  will  always  achieve  a 
specified  bank  angle  in  one  second  (again  for  a  given  stick  force 
gradient)."  Clearly,  both  cannot  be  true,  and  the  data  gathered 
so  far  do  not  allow  us  to  determine  which  (if  indeed  either) 
is  really  correct.  This  is  pointed  out  to  demonstrate  how 
elusive  basic  flying  qualities  parameters  can  be,  and  that  the 
convolution  technique,  with  its  ability  to  simulate  systems 
easily  with  unconventional  or  even  strange  responses,  can  be 
a  valuable  tool  for  testing  hypothesis  such  as  these. 

Our  results  are  somewhat  elementary,  but  unlike  conventional 
handling  qualities  data  they  could  form  a  basis  for  future  study 
of  roll  control  of  highly  augmented  craft,  where  nonclass ical 
responses  may  occur  and  possibly  be  preferred.  (The  regions  of 
relative  pilot  preference  would,  of  course,  first  have  to  be 
redefined  by  "expert"  pilots  into  regions  of  absolute  pilot 
opinion,  most  likely  using  the  Cooper-Harper  pilot  opinion 
scale.)  We  could,  for  instance,  begin  to  explore  how  the  most 
preferred  regions  are  affected  by  variations  from  the  idealized 
response  such  as  "rounding"  of  corners  (i.e.,  nonzero  values  of 
E2  and  E^  in  Fig.  2),  oscillatory  overshoots,  initial  control 
reversals,  and  various  other  control  system  anomalies.  This 
could  be  done  without  regard  to  the  complicated  dynamics  of  the 
aircraft  and  control  system  except  as  depicted  by  the  kinematics 
of  the  response  itself. 
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Grand  Experiment  2  was  designed  as  a  tentative  beginning  to 
such  work  and  attempted  to  determine  the  relative  effects  of  two 
elementary  kinds  of  response  degradations  that  could  result 
from  a  variety  of  airframe/control-system  characteristics.  The 
variables  were  the  and  E2  elements  of  the  roll  rate  step 
response  shown  in  Figure  2.  E^  is  a  pure  time  delay,  and  E2 
can  be  thought  of  as  a  "lagged"  time  delay  or  a  time  delay  with 
a  rounded  corner  similar  to  that  which  is  achieved  by  some 
conventional  lagged  responses.  We  refer  to  the  sura  of  E^  and  E2 
(E^  +  E2)  as  the  "total  effective"  time  delay.  For  these  tests 
E~  was  0.4  second  and  G  was  12  deg/sec/ in.  This  puts  the  base 
configuration  (E^  »  E2  «  0)  within  the  90  percent  region  of 
pilot  preference  (Fig.  12),  but  with  an  Eg  about  0.1  second 
greater  than  the  optimum  value.  E^  was  once  again  zero. 

Figure  15  is  a  three-dimensional  presentation  of  the  data 
which  shows  the  gross  effects  quite  nicely.  Note  that  the  "best" 
and  "worst"  designations  of  this  figure  refer  to  the  configurations 
of  Grand  Experiment  2  only,  and  that  the  pilot  preference  scale 
used  here  is  different  from  the  scale  used  for  the  results  of 
Grand  Experiment  1  (Figures  9,  10  and  11).  For  instance,  remember 
that  the  best  configuration  here  (i.e.,  E^  *  E2  =  0,  E^  ■  0.4 
second  and  E^  *  12  deg/sec/in  .)  is  only  90  to  95  percent  as  good 
as  the  best  configuration  in  Grand  Experiment  1  (see  Figure  12). 

The  most  obvious  result  is  that  increasing  either  E^  or  E2  always 
degraded  pilot  preference.  This  is  not  very  surprising,  and, 
in  fact,  merely  justifies  the  assumption  that  was  made  to  that 
effect  during  the  experimental  design  to  allow  a  reduction  in  the 
size  of  the  experiment  (see  page  3^ .  The  effects  of  both  E^ 
and  E2  are  quite  strong  as  values  of  about  0.1  second  for  either 
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one  cause  almost  a  20  percent  reduction  in  pilot  preference. 

This  is  regarded  as  substantial  because  of  the  pilot's  report 
that  only  the  very  best  configurations  were  really  any  good. 

We  are  really  much  more  interested  in  the  relative  effects 
of  and  E2,  and  these  are  difficult  to  discern  from  Figure  15. 

To  help  clarify  these  effects,  Figure  16  has  been  prepared  to 
show  how  pilot  preference  of  several  representative  Ep  E2 
configurations  changes  when  either  E^  or  E2  is  increased  from 
its  starting  value.  The  alphabetical  labels  indicate  where  the 
starting  Ep  E2  configurations  are  located  on  Figure  15,  and 
the  shaded  areas  surrounding  each  curve  portray  the  standard 
deviation  of  the  data.  The  results  are  not  definitive.  For 
configurations (§)  and @ there  appears  to  be  no  measurable 
difference  between  increasing  E^  or  E2,  as  the  two  curves  never 
separate  beyond  the  standard  deviation  of  the  data.  For  the 
remaining  four  conf igurat ions ©  and ^  however,  there 
does  appear  to  be  a  small  but  measurable  difference  for 
sufficiently  large  increases  in  E^  and  E2*  We  conclude  that 
E^  is  probably  more  detrimental  than  E2 ,  but  the  difference 
between  the  two  effects  is  small,  difficult  to  measure,  and 
generally  not  detectable  for  increases  less  than  about  0.10 
second.  This  does  not  necessarily  mean  that  the  pilots  couldn't 
sense  smaller  differences,  but  rather  that  the  smaller  differences 
(sensed  or  not)  provided  no  perceptible  advantage  or  disadvantage 
in  performance  of  the  tracking  task. 

Figure  17  shows  the  step  responses  for  four  pairs  of 
configurations  that  lie  on  the  first  set  of  curves  in  Figure  16 
at  the  points  labeled  ©,@>  G),  and  respectively.  That  is, 
the  step  response  shown  on  the  left  in  pair  ©  of  Figure  17  has 
the  pilot  preference  indicated  by  the  upper  curve  of  Figure  16a 
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SYMBOLS  A  THROUGH 


Fig.  16  Effect*  of  E1  and  E2  on  Several  representative  Configuration*,  Sheet  1  of  2 
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Fig.  16  Effects  of  and  E2  on  Several  Representative  Configurations, 


at  ©  The  curves  shown  in  Figure  17  are  actual  strip  chart 
recordings  of  time  histories  produced  by  our  real  time  convolu¬ 
tion  routine  in  response  to  step  inputs.  They  portray  the 
differences  in  vehicle  step  responses  that  are  caused  by 
increasing  (left  hand  column  of  responses)  versus  increasing 
E2  (right  hand  column  of  responses),  and  in  association  with 
Figure  16  they  provide  a  feeling  for  the  magnitude  of  these 
differences  needed  to  produce  measurable  differences  in  pilot 
preference. 
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5.  CONCLUSIONS 


A  novel  real-time  simulation  technique  based  on  the  mathe¬ 
matical  concept  of  convolution  has  been  mechanized  on  a  digital 
computer  to  drive  a  single  degree  of  freedom  roll  simulator, 
and  exploratory  roll-control  handling-qualities  studies  have 
been  performed.  The  major  thrust  of  this  effort  was  to 
demonstrate  that  the  convolution  simulation  technique  allows 
researchers  to  study  flying  qualities  in  a  different  way  —  a 
way  that  has  particular  significance  to  the  next  generation  of 
military  aircraft  that  will  have  radical  improvements  in 
maneuverability  due  to  the  addition  of  new  types  of  control 
surfaces  and  the  implementation  of  digital  fly-by-wire  control 
schemes. 

The  flying  qualities  results  discussed  in  Section  4  are 
obviously  rudimentary,  and  many  more  experiments  are  necessary 
to  define  fully  the  preferred  roll  rate  response,  even  in  the 
simple  context  used  here.  The  experimentation  did  show,  however, 
that  direct  manipulation  of  the  kinematics  of  the  pilot's 
cockpit  for  the  purpose  of  defining  handling  qualities  require¬ 
ments  is  feasible.  The  potential  practicality  of  this  technique 
stems  from  the  ever  increasing  complexity  of  the  analytic 
descriptors  of  the  aircraft  and  control  system  responses  and  the 
concurrent  need  to  define  flying  qualities  as  simply  as  possible. 
Researchers  have  often  sought  to  write  flying  qualities 
specifications  as  time  domain  requirements  (see  Refs.  7  and  8), 
but  have  always  done  their  experimentation  with  other  than  time 
domain  parameters.  We  believe  that  this  is  becoming  more  and 
more  impractical,  and  that  the  technique  proposed  here  can  be 
a  very  useful  aid  in  studying  flying  qualities  requirements  of 
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highly  augmented  craft.  Thus  we  think  that  a  long-range  program 
of  intensive  experimentation  should  now  follow,  one  that  uses 
this  new  approach  to  uncover  the  control  responses  that  pilots 
most  prefer.  Then  we  will  be  able  to  use  the  power  of  modern 
control  system  technology  most  effectively  and  provide  the  best 
response  at  each  flight  condition. 

A  particularly  rich  and  timely  application  is  the  study  of 
hover  control  of  VTOL  craft.  The  next  generation  of  these  craft 
should  finally  achieve  operational  status  with  the  military, 
where  success  will  depend  heavily  on  their  flying  qualities  at 
hover.  Current  thinking  has  them  employing  uncoupled,  highly 
augmented  fly-by-wire  schemes  that  will  produce  nonclass ical 
control  responses  at  this  crucial  flight  condition.  This  problem 
is  particularly  amenable  to  study  with  the  convolution  simulation 
technique,  and  we  urge  that  it  be  so  addressed. 
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APPENDIX 

DATA  AND  ANALYSIS 


A- 1 


The  data  from  experiments  1.1  through  1.8  (composing  Grand 
Experiment  i)  are  presented  in  the  proportion  matrices  and  scale 
separation  matrices  of  Tables  A-l  through  A-8.  Also,  the  data 
from  experiments  2.1  through  2.8  (composing  Grand  Experiment  2) 
are  presented  in  the  proportion  and  scale  separation  matrices 
of  Tables  A-9  through  A-16.  All  the  matrices  are  arranged  so 
that  the  configurations  heading  the  columns  are  in  increasing 
order  of  rank  of  their  final  scale  values.  This  was  achieved 
before  the  actual  development  of  the  scale  by  ordering  them 
according  to  the  average  value  of  the  proportion  matrix  columns. 
In  each  table  the  scale  separation  matrix  is  derived  from  the 
data  in  the  proportion  matrix  using  normal  curve  tables  and 
Eq.  (7)  (see  Section  3).  The  final  scale  values  (R^)  were  then 
calculated  at  the  bottom  of  each  scale  separation  matrix  (AR^). 

TEST  OF  INTERNAL  CONSISTENCY 


We  will  demonstrate  the  test  of  internal  consistency 

originally  proposed  by  Mosteller  (Ref.  4)  and  later  demonstrated 

by  Guilford  (Ref.  3)  on  the  data  of  experiment  1.1  shown  in 

Table  A-l.  It  amounts  to  making  a  chi-square  test  of  the 

significance  between  the  P.,  matrix  and  a  matrix  of  proportions 

i  J  ^ 

that  would  be  expected,  P.,  ,  given  the  scale  values  at  the  bottom 

J  k 

of  the  scale  separation  matrix.  Clearly  we  wish  the  test  to  be 

failed  and  thus  indicate  that  there  is  only  an  insignificant 

amount  of  chance  inconsistency  between  the  raw  data  and  the 

scale  produced.  We  start  by  using  the  R^  scale  at  the  bottom 

of  the  AR.,  matrix  (Table  A-l)  to  calculate  the  elements  of  the 

expected  scale  separation  matrix  AR^  in  Table  A-17.  Next  we 

again  use  normal  curve  tables  and  Eq.  (7),  but  this  time  we 
»  ! 

transform  the  AR..  matrix  into  the  P..  matrix  also  shown  in 
jk  Jk 


Table  A~17.  The  P.,  matrix  of  Table  A-l  must  now  be  compared 

J  ^  | 

for  consistency  with  the  P.,  matrix  of  Table  A-17.  Equation 
(A-l)  is  the  formula  for  chi-square  that  applies  to  this 
situation  (Ref.  4). 


N 

821 


£  (»  -  <n2 


(A-l) 


where : 


N  - 


the  statistic  9  ■» 

and  6  * 


number  of  times  each  pair  of  stimuli  are 
judged 

sin'1  yyr 
sin'1  yy^ 


For  the  Table  A-l  data,  N  »  20  and  chi-square  is  calculated  to 
be  4.91.  Reference  4  gives  the  formula  for  the  number  of  degrees 
of  freedom  for  this  situation  as 


df  _  ln-llCn-2), 

2 

Where:  n  number  of  configurations. 

Again  for  the  case  being  studied  here  n  «  5  and  df  «  6.  Reference 
to  a  table  of  chi-square  shows  that  it  takes  a  chi-square  as 
large  as  12.59  to  be  significant  at  the  0.05  level.  We  may  say 
that  the  obtained  chi-square  (4.91)  is  insignificant. 

The  conclusion  to  be  drawn  from  such  a  result  is  that  the 
scale  produced  is  consistent  with  the  data,  and  that  the  assumptions 
made  to  produce  the  scale  (see  page  21  )  have  not  introduced  a 
significant  amount  of  error. 

CURVE  FITTING 

We  recall  that  the  pilot  preference  results  developed  from 
each  pair  comparison  experiment  are  on  an  interval  scale.  Thus 


A-3 


1 


we  are  allowed  to  adjust  all  the  pilot  preferences  from  a  single 
experiment  by  a  constant  amount  without  destroying  the  essential 
character  of  the  results:  the  relative  preference  of  configurations. 
Also,  there  are  many  configurations  in  each  grand  experiment 
that  are  evaluated  in  two  separate  experiments.  For  instance, 
is  at  the  intersection  of  experiments  1.3  and  1.7  (Fig.  6) 
and  is  given  a  scale  value  of  1.185  from  the  data  of  one 
(Table  A-3)  and  another  value,  1.749,  from  the  data  of  the 
other  (Table  A-7).  We  would  like  to  reconcile  these  differences 
and  plot  the  results  of  all  experiments  composing  a  grand  ex¬ 
periment  against  the  same  scale  of  pilot  preference.  (This  is 
allowed  if  we  assume  that  the  term  in  brackets  in  Eq.  (6)  in 
Section  3  is  constant  over  a  grand  experiment. )  We  do  this  by 
selecting  a  set  of  constants  (one  for  each  scale)  that  will 
minimize  the  sum  of  squares  of  all  differences  between  scale 
values  produced  by  two  experiments  for  the  same  configuration 
in  a  grand  experiment  (there  are  15  such  differences  in  Grand 
Experiment  1  and  16  in  Grand  Experiment  2,  as  shown  in  Figures  6 
and  7,  respectively).  To  improve  the  numerics  of  the  least 
squares  solution  process  we  arbitrarily  picked  a  constant  of 
10  for  one  of  the  scales  from  each  grand  experiment.  This  had 
the  effect  of  changing  the  range  of  values  from  0  to  3.0  for  the 
basic  scales  to  10  to  13  for  the  adjusted  scales.  The  basic 
scales  produced  by  each  experiment  are  summarized  (from  Table 
A-l  through  A-16)  in  Tables  A-18  and  A-19  and  the  scales  adjusted 
to  minimize  the  sum  of  squares  of  the  intersect  errors  are 
summarized  in  Tables  A- 20  and  A-21. 

Next  we  made  a  least  squares  fit  of  a  third  order  polynomial 
surface  to  the  adjusted  pilot  preference  data  of  each  grand 
experiment.  The  resultant  polynomials  for  Grand  Experiments  1 
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and  2  are  as  follows. 


Pilot  Preference  *  6.8416  +  2.8026  4-  .9774  G 

J  S  S 

(Grand  Experiment  1)  2  « 

-  6.7805  Eq  +  .0886  EQ  G  -  .0583  G 

3  3  ss  ss 

+  .5644  Eo3  -  .3632  E.2  G  +  .0144  E,G  2 
3  3  ss  3  ss 

+  .00085  G  2  (A-2) 

b  b 

Pilot  Preference  «  12.742  +  .248  E,  -6.271  E2  -  65.468  E  2 
(Grand  Experiment  2)  ?  ? 

-  114.256  EjE2  +  19.482  E2  +  122.332E1 

+  223.923E12E2  +  166.619  E^2 

-  92.999  E23  (A- 3) 

The  goodness  of  fit  of  these  polynomials  to  their  respective  data 
bases  is  indicated  by  the  standard  deviation  of  the  data  from 
the  polynomial  surface,  a,  and  by  the  correlation  coefficient,  p. 

For  the  polynomial  fit  to  the  data  of  Grand  Experiment  1  (Eq.  A-2), 

a  «  0. 1632 

and  p  «  0. 9684 

Likewise  for  thepolynomial  fit  to  the  data  of  Grand  Experiment  2 
(Eq.  A-3), 

a  «  0.1218 
and  p  «  0.9870 

A  more  graphic  portrayal  of  the  goodness  of  fit  is  given  by 
Figures  A-l  and  A-2.  These  show  many  planar  sections  of  the 
polynomial  surfaces  and  the  relationship  of  adjusted  scale  value 
data  to  the  curves.  By  all  standards  considered  the  polynomial 
surfaces  of  Eq.  (A-2)  and  (A-3)  are  good  representations  of  the 
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experimental  results.  Thus,  they  are  considered  to  be  the  final 
fairing  of  the  data,  and  were  used  to  generate  the  pilot 
preference  curves  discussed  in  the  main  body  of  the  report. 
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TABLE  A-  1  THE  PROPORTION  MATRIX  (P..  )  AND  SCALE  SEPARATION 

jk 

MATRIX  (ARjk)  FROM  EXPERIMENT  1.1 


Because  there  are  elements  of  AR.,  missing,  the  AR.  values  are 
computed  as  follows:  k  ^  ^ 

ARj  -  ^  (Rjk'Rj-l,k^n0,  of  k'S 

2 

•  Chi  square  for  the  test  of  internal  consistency:  x  *4.' 


•  For  Rj  to  be  significantly  inconsistent  with  the  data  at 


the  0.05  level  requires  a  chi  square  of  12. 59 


TABLE  A-  2  THE  PROPORTION  MATRIX  (P.,  )  AND  SCALE  SEPARATION 

J  K 

MATRIX  (ARjk)  FROM  EXPERIMENT  1.2 


(N-20) 


.500 

.65 

.  75 

.65 

.90 

.35 

.500 

.90 

1.00 

.80 

.25 

.100 

.500 

.65 

.70 

.35 

0 

.35 

.500 

.65 

.10 

.20 

.30 

.35 

.500 

^R., 

Jk 


0 

.3853 

.6745 

.3853 

1.2816 

-  .3853 

0 

1.2816 

— 

.8416 

-  .  6745 

-1.2816 

0 

.3853 

.5244 

-  .3853 

— 

-  .3853 

0 

.3853 

-1.2816 

-  .8416 

-  .5244 

-  .3853 

0 

.151  .792  .155  .452 

.151  .943  1.098  1.55 


Because  there  are  elements  of  AR.,  missing,  the  AR.  values  are 
computed  as  follows:  .  J 


■  Y.  (Rjk-Rj-i,k>/no-  of  k's 


•  Chi  square  for  the  test  of  internal  consistency:  x  »  4 

•  For  Rj  to  be  significantly  inconsistent  with  the  data  at 
the  0.05  level  requires  a  chi  square  of  12.59 


TABLE  A- 3  THE  PROPORTION  MATRIX  (P.,)  AND  SCALE  SEPARATION 

J  K 

MATRIX  (ARjk)  FROM  EXPERIMENT  1.3 


(N-20) 


VA. 

C11 

C15 

C12 

C14 

C13 

C11 

.500 

.450 

.900 

.850 

1.000 

C15 

.550 

.500 

.650 

.650 

.650 

C12 

.100 

.350 

.500 

.550 

.800 

C14 

.150 

.350 

.450 

.500 

.650 

C13 

0 

.350 

.200 

.350 

.500 

VA 

C11 

C15 

C12 

C14 

C13 

C11 

0 

-  .1257 

1.2816 

1.0364 

— 

C15 

+  .1257 

0 

.3853 

.3853 

.3853 

C12 

-1.2816 

-  .3853 

0 

.  1257 

.8416 

C14 

-1.0364 

-  .3853 

-  .1257 

0 

.3853 

C13 

... 

-  .3853 

-  .8416 

-  .3853 

0 

AR. 

J 

... 

.324 

.396 

.093 

.372 

Rj 

0 

.324 

.720 

.813 

1.185 

Because  there  are  elements  of  AR.^  missing,  the  AR.  values  are 
computed  as  follows:  k  ^ 

ARj  -  ^  (Rjk”Rj_i  k)/no.  of  k' s 

2 

•  Chi  square  for  the  test  of  internal  consistency:  X  «  9. 


For  Rj  to  be  significantly  inconsistent  with  the  data  at 
the  0.05  level  requires  a  chi  square  of  12.59 
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TABLE  A-4  THE  PROPORTION  MATRIX  (P.,  )  AND  SCALE  SEPARATION 

J 

MATRIX  (AR  ,  )  FROM  EXPERIMENT  1.4 


S3 

C16 

C20 

C17 

C19 

C18 

C16 

.500 

.600 

.900 

.850 

.800 

o 

CM 

u 

.400 

.500 

.550 

.650 

.600 

C17 

.100 

.450 

.500 

.600 

.800 

C19 

.150 

.350 

.400 

.500 

.500 

C18 

.200 

.400 

.200 

.500 

.500 

C16 

C20 

C17 

r 

C19 

C18 

C16 

0 

.2533 

1.2816 

1.0364 

.8416 

o 

CM 

u 

-  .2533 

0 

.  1257 

.3853 

.2533 

C17 

-1.2816 

-  .1257 

0 

.2533 

.8416 

C19 

-1.0364 

-  .3853 

-  .2533 

0 

0 

C18 

-  .8416 

-  .2533 

-  .8416 

0 

0 

.580  .165  .273  .052 


.580 

.  745 

1.018 

1.070 

Chi  square  for  the  test  of  internal  consistency:  x  *6.35 
For  Rj  to  be  significantly  inconsistent  with  the  data  at  th« 
0.05  level  requires  a  chi  square  of  12.59 


T 


TABLE  A- 5  THE  PROPORTION  MATRIX  (P.,  )  AND  SCALE  SEPARATION 

J  K 

MATRIX  (AR.,  )  FROM  EXPERIMENT  1.5 


C21 

C22 

C25 

C23 

C24 

C2 1 

.500 

.850 

.  700 

.950 

.900 

C22 

.150 

.500 

.400 

.950 

.850 

C25 

.300 

.600 

.500 

.600 

.  750 

C23 

.050 

.050 

.400 

.500 

.650 

C24 

.100 

.150 

.250 

.350 

.500 

C21 

0 

1.0364 

.5244 

1.6449 

1.2816 

C22 

-1.0364 

0 

.2533 

1.6449 

1.0364 

C25 

-  .5244 

-  .2533 

0 

.2533 

.6745 

C23 

-1.6449 

-1.6449 

-  .2533 

0 

.3853 

C24 

-1.2816 

-1.0364 

-  .6745 

-  .3853 

0 

AR. 

1 

1 

1 

.619 

.147 

.  763 

.044 

Rj 

0 

.619 

.  766 

1.529 

1.573 

Chi  square  for  the  test  of  internal  consistency:  x  *  11. 0< 
For  Rj  to  be  significantly  inconsistent  with  the  data  at  the 
0.05  level  requires  a  chi  square  of  12.59 


Because  there  are  elements  of  AR.,  missing,  the  AR.  values  are 
computed  as  follows:  .  ^ 


ARj  "  Z  ^Rj k~Rj  - 1 , k^no '  of  k’S 


Chi  square  for  the  test  of  internal  consistency:  x 


•  For  Rj  to  be  significantly  inconsistent  with  the  data  at 
the  0.05  level  requires  a  chi  square  of  12.59 


TABLE  A-  7  THE  PROPORTION  MATRIX  (Pjk)  AND  SCALE  SEPARATION 
MATRIX  (AR.,)  FROM  EXPERIMENT  1.7 


SSI 

r 

l23 

00 

I— < 

CJ 

C3 

C8 

C13 

C23 

.500 

.850 

.900 

.850 

.925 

C18 

.150 

.500 

.675 

.  750 

.625 

C3 

.100 

.325 

.500 

.550 

.625 

C8 

.150 

.250 

.450 

.500 

.550 

C13 

.075 

.375 

.375 

.450 

.500 

Vi, 

C23 

C18 

C3 

C8 

C13 

0 

1.0364 

-1.0364 

0 

-1.2816 

-  .4538 

-1.0364 

-  .6745 

-1.4395 

-  .3186 

1.0364 

1.4395 

.6745 

.3186 

.1257 

.3186 

0 

.1257 

-  .1257 

0 

.1049 

.1229 

1.6262 

1. 7491 

Chi  square  for  the  test  of  internal  consistency:  x  ■  9.47 
For  Rj  to  be  significantly  inconsistent  with  the  data  at  the 
0.05  level  requires  a  chi  square  of  12.59 


TABLE  A" 8  THE  PROPORTION  MATRIX  (P^k)  AND  SCALE  SEPARATION 
MATRIX  (AR.,  )  FROM  EXPERIMENT  1.8 


SSI 

C5 

C10 

C25 

C15 

C20 

C5 

.500 

.750 

.700 

.800 

.650 

C10 

.250 

.500 

.550 

.550 

.800 

C25 

.300 

.450 

.500 

.600 

.600 

C15 

.200 

.450 

.400 

.500 

.600 

C20 

.350 

.200 

.400 

.400 

.500 

0 

.671 

.5244 

.8416 

.3853 

CI0 

-  .6745 

0 

.1257 

.1257 

.8416 

C25 

-  .5244 

-  .1257 

0 

.2533 

.2533 

C15 

-  .8416 

-  .1257 

-  .2533 

0 

.2533 

^0 

-  .3853 

-  .8416 

-  .2533 

-  .2533 

0 

ARj 

1 

1 

1 

.401 

.112 

.165 

.153 

Rj 

0 

.401 

.513 

.678 

.831 

Chi  square  for  the  test  of  internal  consistency:  x*"  *  5.01 
For  Rj  to  be  significantly  inconsistent  with  the  data  at  th< 
0.05  level  requires  a  chi  square  of  12.59 


TABLE  A- 9  THE  PROPORTION  MATRIX  (Pjk)  AND  SCALE  SEPARATION 


MATRIX  (AR.,  )  FROM  EXPERIMENT  2.1 


C47 

C46 

.500 

.80 

.20 

.500 

.30 

.35 

.10 

.10 

.500 

.80 

.65 

.80 

.20 

.500 

.65 

.60 

.35 

.35 

.500 

.50 

.20 

.40 

.50 

.500 

46 


.8416 


0 


■  .5244  -  .3853 


■1.2816  -1.2816 


C31  C13 


.5244  1.2816 


.3853  1.2816 


0 


.8416 


.3853 

.8416 

.3853 

.2533 

0 

0 

0 

.4566  .2234  .6542  .1419 


.4566  .6790  1.3332  1.4751  1.556 


Because  there  are  elements  of  AR.,  missing,  the  AR.  values  are 
computed  as  follows:  .  J  J 


(j  -  Y.  <Rjk'Vi,k)/n°-  °£  k's 


Chi  square  for  the  test  of  internal  consistency:  x  *  9.04 
For  Rj  to  be  significantly  inconsistent  with  the  data  at  the 
0.05  level  requires  a  chi  square  of  18.31 


TABLE  A -10  THE  PROPORTION  MATRIX  (Pjk)  AND  SCALE  SEPARATION 
MATRIX  (ARjk)  FROM  EXPERIMENT  2.2 


Because  there  are  elements  of  AR.,  missing,  the  AR.  values  are 
computed  as  follows:  k  ^  ^ 

aRj  '  t  (Rjk-Rj-i,k>/n°-  o£  k's 

2 

•  Chi  square  for  the  test  of  internal  consistency:  x  »  5. 


•  For  Rj  to  be  significantly  inconsistent  with  the  data  at 
the  0.05  level  requires  a  chi  square  of  12.59 


TABLE  A-  11  THE  PROPORTION  MATRIX  (P^k)  AND  SCALE  SEPARATION 
MATRIX  (AR.,  )  FROM  EXPERIMENT  2.3 


(N-40) 


C49 

C43 

C38 

C33 

C2  7 

C49 

.500 

.  700 

.925 

.900 

— 

C43 

.300 

.500 

.800 

.825 

.975 

C38 

.075 

.200 

.500 

.675 

.800 

C33 

.100 

.175 

.325 

.500 

.800 

C27 

— 

.025 

.200 

.200 

.500 

ARjk 


C43 

p 

U38 

C33 

.5244 

1.4395 

1.2816 

0 

.8416 

.9346 

-  .8416 

0 

.4538 

-  .9346 

-  .4538 

0 

-1.9600 

-  .8416 

-  .8416 

.4984 

.8396 

.1684 

.4984 

1.3380 

1.5064 

.8416 


Because  there  are  elements  of  AR.,  missing,  the  AR.  values  are 
computed  as  follows:  .  ^ 


ij  -  Z  <Rjk'Rj-i,k)/no-  °£  k's 


•  Chi  square  for  the  test  of  internal  consistency:  x  »  3. 

•  For  Rj  to  be  significantly  inconsistent  with  the  data  at 
the  0.05  level  requires  a  chi  square  of  12.59 


TABLE  A- 12  THE  PROPORTION  MATRIX  (P.,  )  AND  SCALE  SEPARATION 

J  K 

MATRIX  (AR  )  FROM  EXPERIMENT  2.4 


(N-20) 


Si 

C50 

C44 

C39 

C34 

r 

L28 

C50 

.500 

.65 

.95 

1.00 

— 

C44 

.35 

.500 

.90 

.80 

.90 

C39 

.05 

.10 

.500 

.75 

.80 

C34 

0 

.20 

.25 

.500 

.65 

C28 

— 

.10 

.20 

.35 

.500 

C50 

C44 

C39 

C34 

CO 

OJ 

CJ 

.3853 

1.6449 

0 

1.2816 

-1.2816 

0 

-  .8416 

-  .6745 

-1.2816 

-  .8416 

.6745  .8416 


.3853 


.3600 

.8860 

.3413 

.3444 

.3600 

1.2460 

1.5873 

1.9317 

Because  there  are  elements  of  AR.,  missing,  the  AR.  values  are 
computed  as  follows:  .  ^  J 


-  £  <Rjk'Rj-i,k>/no- o£  k's 


•  Chi  square  for  the  test  of  internal  consistency:  x  ■  6. 

•  For  Rj  to  be  significantly  inconsistent  with  the  data  at 
the  0.05  level  requires  a  chi  square  of  12.59 


TABLE  A- 13  THE  PROPORTION  MATRIX  (Pjk)  AND  SCALE  SEPARATION 


MATRIX  (AR.,  )  FROM  EXPERIMENT  2.5 


29 


.55 


.500 


.45 


.25 


jO 


.500 


.45 


.35 


.10 


C28 

C2  7 

.65 

.90 

.55 

.  75 

.500 

.80 

.20 

.500 

.30 

.45 

.40 

.30 

.70 


.55 


.500 


.35 


.60 


.70 


.65 


.500 


C30 

C29 

C28 

r^- 

Cvj 

O 

C26 

C13 

0 


1257 


.1257  .3853  1.2816 


.1257  .6745 


.3853  -  .1257  0 


.8416  .5244  .2533 


•1.2816  -.6745  -.8416  0 


.1257  .5244 


-  .5744  -  .1257  0 


.3853 


.2533  -.5244  -.3853  0 


2795 

.0860 

.5246 

.0183 

.2245 

2795 

.3655 

.8901 

.9084 

1.1329 

Because  there  are  elements  of  AR.,  missing,  the  AR.  values  are 
computed  as  follows :  k  ^ 

ARj  -  ^  (Rjk‘Rj-1  k^no*  of  k's 

2 

•  Chi  square  for  the  test  of  internal  consistency:  x  "  5.  71 

•  For  Rj  to  be  significantly  inconsistent  with  the  data,  at  the 
0.05  level  requires  a  chi  square  of  18.31 


TABLE  A-  14  THE  PROPORTION  MATRIX  (Pjk)  AND  SCALE  SEPARATION 
MATRIX  (AR.,  )  FROM  EXPERIMENT  2.6 


(N-  20) 


C35 

C34 

C33 

C32 

C31 

C35 

.500 

.80 

.  70 

.90 

— 

C34 

.20 

.500 

.70 

.60 

.75 

C33 

.30 

.30 

.500 

.  70 

.75 

C32 

.10 

.40 

.30 

.500 

.75 

C31 

— 

.25 

.25 

.25 

.500 

AR.. 

Jk 


C34 

C33 

C32 

.8416 

.5244 

1.2816 

0 

.5244 

.2533 

-  .5244 

0 

.5244 

-  .2533 

-  .5244 

0 

-  .6745 

-  .6745 

-  .6745 

.6779 

.0921 

.3031 

.6779 

.  7700 

1.0731 

.6745 


Because  there  are  elements  of  AR.,  missing,  the  AR.  values  are 
computed  as  follows:  .  J  J 


-  £  <vRj-i,k)/no- of  k's 


•  Chi  square  for  the  test  of  internal  consistency:  x  «  5. 

•  For  Rj  to  be  significantly  inconsistent  with  the  data  at 
the  0.05  level  requires  a  chi  square  of  12.59 


TABLE  A-  15  THE  PROPORTION  MATRIX  (PjR)  AND  SCALE  SEPARATION 
MATRIX  (AR.,J  FROM  EXPERIMENT  2 . 7 


(N-  40) 


C40 

C39 

C38 

C37 

C36 

.500 

.725 

.850 

.900 

1 

1 

1 

.2  75 

.500 

.775 

.825 

.850 

.150 

.225 

.500 

.  750 

.875 

.100 

.175 

.250 

.500 

.750 

1 

1 

1 

.150 

.125 

.  ?50 

.500 

AR.. 

Jk 


.5978  1. 0364  1.2816 


.5978  0 


.7554  .9346  1.0364 


-1.0364  -  .7554  0 


.6745  1.1503 


-1.2816  -  .9346  -  .6745  0 


.6745 


-1.0364  -1.1503  -  .6745  0 


.4559 

.4191 

.4498 

.4817 

.4559 

.8750 

1.3248 

1.8065 

Because  there  are  elements  of  AR.,  missing,  the  AR.  values  are 
computed  as  follows:  ^  ^  ^ 

ARj  «  J  (Rjk"Rj ^ /no.  of  k's 

2 

•  Chi  square  for  the  test  of  internal  consistency:  x  »  8. 

•  For  Rj  to  be  significantly  inconsistent  with  the  data  at 
the  0.05  level  requires  a  chi  square  of  12.59 


TABLE  A-  16  THE  PROPORTION  MATRIX  (Pjk)  AND  SCALE  SEPARATION 
MATRIX  (AR.,  )  FROM  EXPERIMENT  2.8 


:  59 

C45 

C44 

C43 

°42 

C41 

C45 

.500 

.80 

.80 

.90 

— 

C44 

.20 

.500 

.75 

.95 

.95 

C43 

.20 

.25 

.500 

.50 

.90 

C42 

.10 

.05 

.50 

.500 

.75 

C41 

— 

.05 

.10 

.25 

.500 

C44 

C43 

CM 

.8416 

.8416 

1.2  816 

0 

.6745 

1.6449 

-  .6745 

0 

0 

-1.6449 

0 

0 

-1.6449 

-1.2816 

-  .6745 

.3717 

.6714 

.4035 

.3717 

1.0431 

1.4466 

1.2816 


.6745 


Because  there  are  elements  of  AR.,  missing,  the  AR.  values  are 
computed  as  follows:  ^  ^  ^ 

AR.  «  J  (Rj k.” Rj - i , k> /n° '  of  k’s 

2 

•  Chi  square  for  the  test  of  internal  consistency:  x  *  7. 

•  For  Rj  to  be  significantly  inconsistent  with  the  data  at 
the  0.05  level  requires  a  chi  square  of  12.59 
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TABLE  A- 18  SUMMARY  OF  BASIC  PILOT  PREFERENCE  SCALES  PRODUCED  BY 
GRAND  EXPERIMENT  1  (NO  CONSTANTS  ADDED) 


Note:  Units  of  Pilot  Preference  are 


TABLE  A- 19  SUMMARY  OF  BASIC  PILOT  PREFERENCE  SCALES  PRODUCED 
BY  GRAND  EXPERIMENT  2  (NO  CONSTANTS  ADDED) 


Note:  Units  of  Pilot  Preference  are  o. 


>< 

a 

w 

o 

8 

:=> 

Q 

Q 

o 

W 

os 

Q 

fa 

Q 

C 

CO 

CO 

g 

CO 

CO 

W 

a 

£  8 

Bag 

SKS 

g  s  W 

fc  N  H 
H  H  W 

sis 


‘Si 

fa  W  W 

O  fa 

><  w 
5h  W  N 


to  O  S 


Note:  Units  of  Pilot  Preference  are  a9  -\/2(l-r 
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Fig.  A-2  Planar  Sactiom  of  tha  Polynomial  Surfaca  Fit  to  tha  Adjuttad  Data  of  Grand  Exparimant  2 
Com  pa  rad  to  tha  Data  (Shaat  1  of  2) 


Fig.  A-2  Planar  Sectiont  of  the  Polynomial  Surface  Fit  to  the  Adjutted  Date  of  Grand  Experiment  2 
Compared  to  the  Date  (Sheet  2  of  21 


